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ABSTRACT 


Early  in  their  career,  submarine  officers  must  develop  and  maintain  many  skills  to 
qualify  as  periscope  operators.  This  critical  position  comprises  complex  tasks,  requiring 
precise  steps,  accurate  mental  calculations,  and  celerity  of  actions  and  thoughts,  while 
obeying  specific  rules  to  ensure  the  safety  of  the  submarine  and  its  crew.  Currently,  gaps 
in  training  exist  between  the  professional  courses  a  submarine  officer  attends  over  a 
career  to  learn  and  practice  these  skills.  Following  an  instructional  system  design  process, 
this  thesis  developed  a  3D,  game -based  periscope  tactical  training  prototype.  This  part- 
task  training  solution  can  provide  feedback  on  predetermined  performance  measures, 
either  automated,  real-time,  or  post-hoc.  The  second  part  of  this  thesis  used  the  prototype 
environment  to  investigate  the  impact  of  feedback  type  on  performance.  Results  indicated 
that  participants  receiving  real-time  feedback  had  significantly  better  performance  than 
those  who  received  post-hoc  feedback  during  an  initial  trial.  Performance  for  both  groups 
significantly  improved  for  subsequent  trials;  by  the  third  trial,  however,  significant 
differences  were  not  observed  for  feedback  type.  Additionally,  results  of  an  exploratory 
analysis  showed  feedback  type  affected  performance  by  age,  time  in  service,  and  gaming 
experience.  Results  of  this  thesis  support  the  use  of  game-based  simulation  as  training 
tools  and  that  feedback  type  could  be  tailored  to  individuals  based  on  factors  such  as  age 
and  experience  to  optimize  training  outcomes. 
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I.  INTRODUCTION 


A.  THESIS  STATEMENT 

A  submarine  officer  (SO)  must  develop  and  maintain  a  myriad  of  skills  over  the 
course  of  a  successful  career.  A  particularly  challenging  set  of  these  skills  are  those 
required  of  an  officer  to  become  qualified  as  a  periscope  operator.  Often  considered  the 
most  important  sensor  on  a  submarine,  the  periscope  allows  crews  to  collect  crucial 
information  that  impact  on  safety  and  effectiveness  of  operations  of  the  ship.  Being 
qualified  as  periscope  operator  requires  the  ability  to  master  complex  tasks,  which  require 
precise  steps,  accurate  mental  calculations,  and  celerity  of  actions  and  thoughts,  while 
recalling  and  obeying  specific  rules  that  ensure  the  safety  of  the  submarine  and  its  crew. 

Currently,  the  only  options  for  hands-on  training  are  through  participation  on 
exercises  aboard  submarines  or  scheduled  training  sessions  in  a  full-scale  simulator 
called  attack  trainer  (AT). 

This  thesis  proposes  the  use  of  game-based  simulation  for  training  and  practicing 
periscope  tasks.  To  this  end,  a  high-fidelity  three-dimensional  (3D)  environment  called 
Submarine  Tactical  Training  System  (STTS)  was  created  with  the  goal  of  closing  training 
gaps  that  exist  between  career  courses  for  SOs  and  limited  opportunities  to  participate  in 
AT  training  sessions  and  exercises  on  submarines.  Additionally,  we  conducted  an 
evaluation  of  game-based  feedback  during  the  training  session  compared  to  process- 
based  diagnostic  feedback  provided  after  the  end  of  the  task  using  the  STTS. 

B.  MOTIVATION 

The  knowledge,  skills  and  abilities  (KSA)  necessary  for  the  proper  operation  of 

the  periscope  are  acquired  during  courses  spread  across  the  SO’s  career.  Section  II. A 

provides  background  information  about  these  career  courses.  Concepts  learned  during 

one  course  are  required  for  all  subsequent  courses  and  retaining  the  KSAs  developed  in 

those  courses  is  the  responsibility  of  the  individual  officer.  When  assigned  to  duties  away 

from  submarines  or  assigned  to  submarines  in  long  maintenance  periods,  the  AT  is  the 

only  training  solution  available.  However,  individual  training  opportunities  with  ATs  are 
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limited  because  their  schedules  are  normally  filled  with  planned  courses,  preventive 
maintenance,  and  preparation  for  future  courses. 

The  ubiquity  of  powerful  personal  computers  and  SOs  familiarity  with  computer 
games  suggest  that  game-based  training  systems  should  be  explored  as  a  solution  for  the 
existing  training  gap.  According  to  Michael  and  Chen  (2005): 

Advances  in  computer  hardware  and  software  allow  games  to  move  into 
other  aspects  of  education  and  training.  Many  skills  can  be  taught  only  by 
doing,  and  many  lessons  can  be  learned  only  through  failure.  Serious 
games  allow  training  to  occur  in  a  non-lethal  environment.  Routinely, 
soldiers  enter  complex,  highly  detailed  simulations  with  3D  rendering  and 
real-world  physics  calculations  to  practice  their  military  training,  (p.  XVI) 

Furthermore,  these  types  of  systems  can  be  developed  to  allow  students  to  train  at 
their  own  pace,  an  ideal  feature  for  SOs  with  busy  schedules.  However,  the  self-regulated 
nature  of  the  proposed  solution  demands  providing  trainees  with  feedback  to  avoid 
harmful  effects  of  negative  training  (Buff  &  Campbell,  2002).  This  thesis  consists  of  two 
main  phases.  First,  following  an  instructional  system  design  (ISD)  process  this  thesis 
developed  a  game-based  SO  training  prototype.  Next,  this  thesis  conducted  an 
investigation  to  determine  if  different  feedback  types  could  increase  training 
effectiveness  of  the  prototype. 

C.  RESEARCH  QUESTIONS  (RQ) 

1.  Can  a  game-based,  part-task  periscope  simulation  that  incorporates  the 
appropriate  cues  to  train  identified  knowledge,  skills,  and  abilities  be 
developed  to  supplement  periscope  operator  training? 

2.  Do  differences  in  trainee  performance  exist  when  exposed  to  different 
feedback  types  over  the  course  of  multiple  sessions? 

3.  What,  if  any,  differences  exist  between  the  real-time  feedback  (RTF)  or 
post-hoc  feedback  (PHF)  on  final  trainee  performance? 
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D.  HYPOTHESES 


A  series  of  hypotheses  were  developed  to  investigate  the  research  questions  stated 
above.  These  include: 

(1)  Following  an  ISD  process,  the  development  of  a  game-based  trainer,  is 
addressed  in  Chapter  IV. 

(2)  Hlo:  Participants  who  receive  PHF  will  perform  the  same  as  those  who 
receive  RTF  on  the  STTS  task  over  repeated  trials. 

//2a:  Participants  who  receive  PHF  will  have  significant  performance 
differences  when  compared  to  those  who  receive  RTF  on  the  STTS  task 
over  repeated  trials. 

(3)  H3q.  Participants  who  receive  PHF  will  perform  the  same  as  those  who 
receive  RTF  on  the  STTS  task. 

//3a:  Participants  who  receive  PHF  will  have  significant  performance 
differences  when  compared  to  those  who  receive  RTF  on  the  STTS  task. 

E.  SCOPE 

This  thesis  covers  only  basic,  unclassified  periscope  techniques.  Some  details  of 
these  techniques  are  different  in  each  navy  and  most  of  their  practical  applications  are 
treated  as  classified  information,  which  were  omitted  to  avoid  classification  issues.  This 
thesis  will  focus  only  on  the  basic  unclassified  definitions  and  concepts  from  the 
techniques,  which  are  explained  in  the  section  II.B.l.  Section  UFA  describes  the  process 
used  to  select  the  appropriate  tasks  and  content  to  be  covered  without  breaching  classified 
information. 
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II.  BACKGROUND 


The  stealth  provided  by  operating  underwater  enables  submarines  to  pose  a 
considerable  threat  to  enemy  surface  ships.  An  opposing  force  must  commit  a  substantial 
amount  of  anti-submarine  warfare  (ASW)  resources  to  sweeping  vast  areas  of  the  ocean 
to  counter  a  single  submarine,  accept  a  high  level  of  risk  to  operate  in  an  area  where 
submarine  presence  is  possible,  or  be  forced  to  avoid  areas  entirely.  These  are  some  of 
the  characteristics,  which  make  the  submarine  a  valuable  anti-access/area-denial  asset. 

A  submarine  is  most  effective  when  it  is  able  to  remain  hidden.  When  running  in 
deep  waters,  submarines  are  virtually  undetectable  except  by  very  capable  ASW  forces, 
which  provide  safety  against  attack.  Additionally,  submarines  normally  operate 
submerged  at  depths  significantly  greater  than  the  draft  of  all  surface  ships.  Therefore, 
once  at  depth,  submarines  are  effectively  immune  to  collisions  with  surface  ships. 

However,  some  situations  force  submarines  to  operate  at  periscope  depth  (PD), 
including  performing  visual  searches,  communications,  recharging  batteries,  conducting 
navigational  fixes,  ventilating,  and  preparing  to  surface.  Depending  on  the  task,  the 
submarine  is  forced  to  expose  one  or  more  of  its  masts.'  In  these  situations,  the  advantage 
of  stealth  is  compromised  and  a  submarine  can  be  vulnerable  to  visual  and  radar 
detection.  Likewise,  while  at  PD,  the  submarine’s  depth  is  less  than  the  draft  of  surface 
ships  and  the  danger  of  colliding  with  one  exists.  To  address  these  risks,  submarines  must 
follow  a  set  of  procedures,  called  periscope  depth  operation  techniques  (PDOT),  to 
minimize  exposure  to  ASW  forces  and  prevent  collisions,  while  assuring  the  completion 
of  all  the  required  tasks. 

In  order  to  understand  this  research,  additional  background  information  is 
required  in  three  areas.  The  first  two,  current  Brazilian  SO  training  and  periscope  safety, 
deal  specifically  with  the  Brazilian  Navy,  while  the  third  area  is  feedback  research  from 
instructional  research  literature. 


^  Most  submarines  perform  different  tasks  using  multiple  masts,  such  as  a  periscope,  communications 
mast,  snorkel,  radar,  electronic  support  measures  (ESM),  among  others. 
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A. 


CURRENT  BRAZILIAN  SUBMARINE  OFFICER  TRAINING 


Over  the  course  of  a  career,  Brazilian  SOs  develop  KSAs  that  are  required  for  the 
different  phases  of  their  profession.  KSAs  are  formally  taught  in  a  series  of  specialized 
courses  that  are  attended  as  the  SOs  transition  between  the  phases  of  their  careers.  After 
successful  completion  of  each  course,  a  SO  is  qualified  to  perform  new  duties  on  board 
the  submarine.  As  a  career  progresses,  the  SO  responsibilities  graduate  from 
technical/platform-control  tasks  to  more  tactically  relevant  duties.  Regardless  of  phase, 
each  of  the  formal  courses  consists  of  three  main  parts:  (1)  theoretical  instruction  in 
classrooms,  (2)  hands-on  practice  in  a  full-scale  simulator,  and  (3)  a  final  practical  phase 
aboard  a  submarine  at  sea. 

The  first  course  SOs  participate  in  is  the  Submarine  Officer  Specialization  Course 
(SOSC).  This  course  is  required  for  all  SOs  prior  to  joining  the  submarine  fleet.  SOSC 
focuses  on  the  operation  and  maintenance  of  the  submarine  as  a  platform.  Successful 
completion  of  the  SOSC  qualifies  the  SO  to  serve  in  the  maneuvering  compartment 
where  his  duties  include  maintaining  course,  depth,  and  speed  as  ordered  by  the  Officer 
of  the  Deck  (OOD).  Upon  completion  of  SOSC  classroom  instruction,  students  practice 
skills  in  a  diving  simulator.  During  practice  runs  in  the  simulator,  an  extensive  focus  is 
placed  on  emergency  procedures  in  order  to  allow  trainees  to  gain  experience  without  the 
potential  consequences  of  failure  that  exist  aboard  a  real  submarine.  After  completion  of 
SOSC,  trainees  report  to  their  first  submarine  where  they  become  qualified  to  stand  the 
watch  and  are  able  to  practice  techniques  they  learned  in  SOSC  while  performing  their 
normal  duties. 

Preparation  for  the  Duty  of  Periscope  and  Sound  (PDPS)  is  the  second  career 
course  a  SO  attends  during  his  career.  This  course  marks  the  transition  from  maneuver 
officer  to  OOD.  Learning  objectives  in  this  second  course  focus  on  maintaining  the 
submarine’s  safety.  The  primary  task  trained  is  how  to  position  properly  a  submarine  to 
return  to  PD  using  passive  sonar  signals  as  input.  During  PDPS,  students  practice  these 
tasks  in  the  AT.  The  AT  is  the  simulator  SOs  will  use  during  all  remaining  career  courses 
and  any  other  simulation  training  events  they  may  receive.  Although  this  course  contains 

an  introduction  to  PDOT,  those  KSAs  are  taught  extensively  in  the  next  course. 
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The  third  course  a  SO  takes  is  Submarine  Operations  for  Officers  (SOO),  which 
qualifies  them  to  become  a  department  head  on  a  submarine.  This  course  covers  all 
periscope  techniques  and  focuses  on  submarine  warfare.  Students  learn  to  plan  and 
execute  a  range  of  different  submarine  operations,  from  anti-surface  warfare  to  secondary 
tasks  like  visual  and  electronic  reconnaissance  and  deployment  of  special  operation 
forces.  SOO  uses  the  AT  extensively  to  prepare  students  for  performing  on  ships  and  to 
minimize  the  shipboard  portion  of  the  course,  since,  unlike  the  previous  courses,  SOO 
requires  one  or  more  surface  combatant  ships  to  support  the  tactical  missions  as  an 
opposing  force.  The  onboard  portion  of  SOO  is  very  costly,  and  it  is  imperative  that 
students  are  prepared  prior  to  the  at-sea  opportunity  because  that  portion  of  SOO  is 
generally  reserved  for  final  exam  purposes  instead  of  training  events. 

To  be  assigned  commanding  officer  of  a  submarine,  a  SO  has  to  pass  the 
Submarine  Commanding  Course  (SMCC)  successfully.  This  is  the  final  and  most 
challenging  of  the  courses  in  an  SO’s  career,  and  is  largely  based  upon  the  British  Royal 
Navy’s  Perisher  course.  According  to  Mack’s  (2003,  p.  2)  description  of  the  Perisher 
course,  “ranging  and  estimating,  mental  agility,  overall  situational  awareness,  command 
presence,  intellectual  honesty,  safety,  character  under  pressure”  are  some  of  the  skills 
required  to  succeed  in  SMCC.  At  this  point,  there  are  very  few  new  topics  and  the 
classroom  phase  is  dedicated  to  refresher  training.  However,  each  prospective 
commanding  officer  gets  significant  time  in  the  AT  before  going  to  sea,  where  he  will  be 
required  to  demonstrate  that  he  has  mastered  the  KSAs  learned  over  the  course  of  his 
career  in  order  to  earn  command. 

These  courses  are  attended  as  a  SO  rises  in  rank  from  ensign  to  commander  over 
the  course  of  ten  to  fifteen  years.  There  is  significant  time  between  each  course  and 
maintaining  the  KSAs  acquired  in  them,  which  are  also  required  for  the  next,  is  the 
responsibility  of  the  SO  (Bastos,  2014).  As  careers  progress,  SOs  are  often  assigned  to 
duties  away  from  submarines.  If  an  SO  is  assigned  to  a  unit,  which  is  geographically 
close  to  the  Brazilian  Navy  Submarine  School  (BNSS),  he  may  conduct  refresher  training 
in  the  AT  so  it  is  possible  to  maintain  his  skills.  However,  even  if  an  officer  is  located 
close  to  the  BNSS,  the  AT  is  often  unavailable  to  them  because  it  is  being  used  for 
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scheduled  courses,  undergoing  preventive  maintenance,  being  prepared  for  future 
courses,  or  recovering  from  unexpected  equipment  failures  thus  limiting  an  individual 
SO’s  access  for  skill  maintenance.  Additionally,  the  use  of  the  AT  for  individual,  non- 
scheduled,  training  sessions  may  not  be  approved  due  to  the  increased  maintenance  and 
personnel  costs  involved  in  running  the  system. 

During  all  the  three  career  courses,  an  experienced  instructor  closely  observes 
trainees.  During  live  events,  if  a  trainee  is  operating  a  periscope,  an  instructor  uses  the 
other  periscope  to  continuously  ensure  the  safety  of  the  submarine.  While  operating  in 
this  tandem,  it  is  commonplace  for  instructors  to  provide  feedback  to  the  student  about 
their  observations  during  the  execution  of  an  exercise.  This  feedback  is  given 
immediately  following  the  execution  of  a  task.  For  the  purposes  of  this  thesis,  this  type  of 
feedback  is  defined  as  real-time  feedback  (RTF).  The  most  common  examples  of  RTF  are 
the  corrections  about  estimations  and  measurements  made  by  the  student.  In  the 
simulators,  there  is  also  an  experienced  instructor  who  has  access  to  ground  truth  while 
operating  the  instructor  console.  In  this  case,  the  instructor  also  provides  RTF.  The 
amount  of  feedback  provided  depends  on  the  instructor’s  judgement  and  trainee’s 
performance,  varying  from  constant  interventions  to  none  at  all. 

Another  method  to  informing  students  about  performance  is  when  an  instructor 
takes  note  of  the  errors  committed  during  a  run  and  communicates  them  to  the  student 
after  the  completion  of  the  task.  For  this  thesis,  this  type  of  feedback  is  defined  as  post- 
hoc  feedback  (PHF). 

B.  PERISCOPE  SAFETY 

Performing  PDOT  involves  a  trade-off  between  acquiring  important  information 
and  exposition.  The  amount  of  time  that  the  observation  takes  is  directly  proportional  to 
quantity  and  quality  of  information  that  the  periscope  officer  (PO)  is  able  to  obtain,  but 
also  proportionally  increases  the  risk  of  counter-detection  either  visually  or  by  radar.  The 
main  goal  of  the  periscope  safety  procedures  is  to  ensure  the  crew  operates  at  the  optimal 
ratio  between  information  gathering  and  time. 
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There  are  three  main  purposes  related  to  periscope’s  use:  navigation,  surveillance, 
and  tactical  employment.  The  navigation  task  is  normally  done  by  triangulating  three  or 
more  bearings  obtained  from  known  navigation  aids.  It  is  also  possible  to  get  distances 
from  reference  points  for  which  the  height  is  known.  A  relatively  accurate  position  can  be 
determined  with  only  one  observation  of  a  reference  point  by  taking  its  bearing  and 
distance  at  the  same  observation.  The  navigation  task  will  not  be  in  the  scope  of  this 
thesis. 

The  employment  of  the  periscope  for  surveillance  is  accomplished  through 
careful,  360-degree  observation  around  the  submarine.  When  operating  at  PD,  a 
submarine  is  exposed  to  the  risk  of  collision  with  surface  ships,  making  surveillance  a 
critical  task.  However,  the  deliberate  use  of  the  periscope  for  surveillance  incurs  an 
increased  risk  of  counter-detection.  Therefore,  submariners  follow  a  careful  set  of  rules 
defined  to  maximize  safety  while  minimizing  exposure  of  the  periscope  mast  in  order  to 
reduce  the  counter-detection  risk.  These  rules  and  tactical  concepts  of  employment  are 
discussed  in  the  following  sections. 

C.  PERISCOPE  DEPTH  OPERATION  TECHNIQUES 

Originally,  the  periscope  technique  arose  from  the  need  to  fire  unguided  straight 
running  torpedoes  against  surface  targets  using  no  other  data  besides  images  obtained  by 
using  the  periscope.  Even  though  today  the  likelihood  of  successfully  performing  such 
attacks  is  extremely  low,  the  mastery  of  these  techniques  remains  a  requirement  in  many 
modern  navies  because  they  are  useful  in  many  other  situations  (Bastos,  2014).  For 
example,  these  techniques  are  considered  essential  for  the  training  of  a  submarine 
commanding  officer,  who  must  know  how  to  position  the  submarine  relative  to  any 
contact  of  interest,  using  only  stopwatches  and  agile  mental  calculations  (White,  1993). 

When  operating  at  PD,  submarine  crews  track  all  contacts  present  in  the  visual 
spectrum  within  visual  range.  Once  a  contact  is  detected  and  tracked,  the  PO  must 
determine  its  bearing  and  heading  and  constantly  update  the  contact’s  range,  in  order  to 
reserve  enough  time  to  properly  maneuver  and  to  avoid  sailing  in  dangerous  situations. 
The  concepts  presented  in  this  chapter  are  a  basic,  unclassified  description  of  the  rules 
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that  allow  a  PO  to  maintain  the  safety  of  the  submarine  and  crew  while  maximizing  the 
chances  of  successfully  complete  a  mission. 

Some  detailed  periscope  techniques  aspects  can  vary  between  navies  and  are 
usually  found  in  classified  manuals.  This  thesis  will  use  basic  unclassified  periscope 
concepts  from  the  technique. 

a.  All-Round  Look 

The  all-round  look  (ARL)  is  the  periscope  technique  related  to  the  surveillance 
task.  This  observation  is  used  to  find  new  contacts  and  obtain  situation  awareness.  It  is 
also  important  to  observe  the  environment  and  report  any  changes  in  the  environment, 
like  visibility  and  rain  (Ferguson,  2014).  When  conducting  the  ARL,  movement  of  the 
periscope  is  continuous  around  the  360°,  without  pauses,  using  a  high  field  of  view 
(zoomed  out),  and  correct  lens  elevation  (sight  picture  of  1/3  sea,  2/3  sky).  The  ideal 
rotation  speed  while  doing  the  ARL  is  such  that  it  is  not  too  slow,  to  avoid  counter¬ 
detection,  and  not  too  fast,  to  assure  the  accuracy  of  the  observation  and  the  detection  of 
any  visible  contact  (“Perisher,”  2007).  The  minimum  and  maximum  time  limits  allowed 
for  this  observation  can  vary  from  navy  to  navy. 

b.  Safe  Depth 

Safe  depth  (SD)  can  be  defined  as  the  minimum  depth  at  which  a  submarine 
should  dive  when  close  to  a  surface  vessel.  At  this  depth,  there  is  sufficient  separation 
from  the  keel  of  the  surface  ship  and  the  top  of  the  submarine’s  sail  (NATO,  2002).  This 
separation  must  be  increased  when  operating  under  heavy  weather  or  high  sea  states  since 
the  keel  of  a  ship  may  go  deeper  than  its  normal  draft.  When  determining  SD  the  type  of 
surface  ships  expected  to  be  found  in  the  area  of  operation  must  be  considered.  The  draft 
of  these  ships  will  contribute  to  SD  calculations.  Additionally,  the  height  of  the 
submarine  (distance  from  keel  to  top  of  the  sail)  and  the  desired  separation  between  the 
submarine  and  surface  ship  are  part  of  this  calculation.  An  example  of  these  factors  is 
shown  in  Figure  1. 
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Figure  1.  Safe  Depth. 


c.  Go-Deep  Range  /  Go-Deep  Circle 

An  important  goal  of  operating  a  submarine  is  to  ensure  that  the  submarine  can 
reach  SD  prior  to  a  surface  ship  reaching  the  submarine’s  location.  Each  type  of 
submarine  takes  some  specific  amount  of  time,  reported  in  seconds,  to  dive  from  PD  to 
SD  (TSD)  and  has  a  specific  speed  during  the  decent  (SpeedSub).  Varying  the  SD  will 
vary  TSD  proportionally.  Go-deep  range  (GDR)  is  the  distance  that  a  surface  ship  will  be 
able  to  travel,  relative  to  the  submarine,  in  TSD  seconds.  Because  a  crew  wants  to  ensure 
that  the  submarine  will  be  at  SD  before  relative  motion  causess  a  surface  ship  and 
submarine  to  occupy  the  same  location  at  the  same  time,  the  ship’s  maximum  speed 
(MaxSpeedtarget)  is  used  when  calculating  GDR.  Equation  (1)  is  used  to  calculate 
the  GDR.^ 


GDR{yards^  = 


(Speed^^i,  +  MaxSpeed^^^gJx  100 
180 


xT 


SD 


(1) 


The  go-deep  circle  (GDC)  is  a  circle  centered  on  the  submarine,  with  a  radius  of 
GDR  yards  (“Perisher,”  2007).  The  GDC  is  considered  the  submarine’s  safe-heaven.  A 
general  rule  of  PDOT  is  to  dive  to  SD  prior  to  any  surface  ship  entering  the  GDC  (Eigure 
2-a),  which  ensures  that  the  submarine  will  be  at  SD  prior  to  the  surface  ship  passing 
overhead  (Eigure  2-b). 


2  Throughout  this  thesis,  all  distances  are  in  yards,  all  times  are  in  seconds  and  all  speeds  are  in  knots, 
unless  otherwise  noted. 
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a.  The  surface  ship  reaches  the  GDC;  the  submarine  starts  descending. 

b.  The  submarine  reaches  the  SD  safely  below  the  surface  ship  before  it  reaches  the  sub. 

Figure  2.  Go-Deep  Circle. 


Although  this  method  for  GDR  calculation  is  the  safest  for  the  submarine,  it 
comes  with  a  cost  in  the  flexibility:  unless  the  surface  ship  is  directly  ahead  of  the 
submarine  and  heading  straight  for  it,  the  GDR  will  be  greater  than  need  be,  and  the 
submariner  will  have  to  dive  earlier  than  actually  required.  A  more  detailed  GDR 
calculation  can  be  accomplished  by  dividing  the  circle  around  the  submarine  into  sectors. 
When  a  contact  is  approaching  from  a  different  relative  bearing  (0),  the  submarine’s 
component  of  the  relative  motion  can  be  estimated  more  accurately.  This  is  shown  in 
Equation  (2). 


GDRYyards^  = 


[{Speed X cos 0)  +  MaxSpeed^^^^^,]x  100 
180 


xT 


SD 


(2) 


The  exact  angles  used  to  reduce  the  GDC  by  sectors  are  a  specific  decision  made 
by  each  navy  and  are  normally  classified  information. 


d.  All  Round  Look  Interval 

ARE  Interval  is  the  “acceptable  time  interval  between  two  AREs”  when  the 
“worst  case  scenario  is  considered:  a  new  contact  appears  just  after  the  periscope  looks 
toward  its  direction”  (Bastos,  2014,  p.  16).  ARE  assumes  that  the  surface  contact 
approaches  with  its  maximum  speed  and  is  on  a  collision  course.  The  ARE  interval  must 
be  such  that  is  always  smaller  than  the  time  that  a  contact  will  take  to  reach  the  GDR. 

12 


This  assures  that  all  contacts,  even  in  the  worst-case  scenario,  will  always  be  detected 
before  reaching  the  GDR. 


e.  Look  Interval 


When  a  contact  is  observed  and  its  range  determined,  and  after  the  periscope  is 
lowered,  a  PO  must  again  consider  the  worst-case  scenario  based  on  the  information  that 
he  has  collected.  This  case  usually  means  considering  that  the  surface  contact  has 
instantaneously  turned  towards  the  submarine  and  has  increased  to  its  maximum  speed. 
Based  on  this  worst-case  scenario,  the  look  interval  (LI)  is  defined  as  the  time  required 
for  that  individual  ship  to  get  from  its  present  position  to  the  GDR.  Immediately  after  the 
periscope  is  lowered,  the  PO  must  calculate  the  LI  and  before  this  time  is  reached,  the  PO 
must  determine  the  contact’s  range  again  to  ensure  that  it  is  further  than  the  GDR 
(“Perisher,”  2007). 


Each  contact  in  the  visual  range  must  have  its  LI  calculated  and  controlled.  If  the 
LI  expires  before  the  vessel  is  ranged  again,  emergency  procedures  require  the  PO  to 
order  an  urgent  dive  to  SD.  The  calculation  to  determine  LI  is  shown  in  Equation  (3). 


LI 


{s,Qconds'\ 


Rgf^getarget-GDR  180 

MaxSpeed,^^^^,  +  Speed  100 


(3) 


/.  Angle-on-the-Bow 

The  angle-on-the-bow  (AOB)  is  defined  as  the  relative  bearing  of  the  submarine 
when  observed  by  the  target.  It  can  be  compared  to  what  the  surface  sailors  usually  call 
target  angle,  but  with  the  difference  that  AOB  must  include  the  side  of  the  target  and  the 
angle  is  between  0°  and  180°,  while  target  angle  is  simply  the  clockwise  angle  from  the 
target’s  bow.  The  difference  between  AoB  and  target  angle  is  illustrated  in  Eigure  3. 
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Figure  3.  Angle-on-the-Bow  and  Target  Angle  Compared. 
g.  Contact  Observation 

The  contact  observation  (CO)  is  the  technique  used  when  observing  the  contact 
for  the  first  time  or  when  maneuvering  close  to  a  contact  and  tracking  it  with  a 
stadimetric  plot.  It  is  designed  to  obtain  the  maximum  amount  of  information  from  a 
contact  in  a  minimum  time,  and  is  always  done  in  the  same  order  to  avoid  confusion. 
During  a  CO,  the  PO  should  mark  the  contact’s  bearing,  range  (by  the  use  of  stadimeter 
built  into  the  periscope  or  other  ranging  system),  and  after  lowering  the  periscope,  inform 
the  AOB,  and  mentally  calculate  the  LI  based  on  the  range  observed.  Table  1  illustrates 
this  sequence  and  phraseology  for  a  contact  observation. 


Table  1.  Contact  Observation  Phraseology. 


“Observation  -  contact  03” 

State  type  of  observation  and  the  contact  to  observe 

“Raise  the  one” 

Order  to  raise  the  appropriate  periscope 

“Bearing  030” 

Report  bearing  reading 

“Distance  2300  yards” 

Report  distance  reading 

“Lower” 

Order  to  lower  the  periscope 

“Angle-on-the-bow,  Port  60” 

Report  side  and  value  estimation  for  AoB 

“Look  Interval,  1  minute  and 
20  seconds” 

Report  the  mentally  calculated  look  interval  for  the 
observed  distance. 

14 


The  time  that  a  contact  observation  lasts  should  be  as  short  as  possible  but  the 
maximum  limit  allowed  can  vary  depending  on  the  country. 

D.  TYPES  OF  FEEDBACK 

The  role  that  feedback  plays  in  the  learning  process  is  a  controversial  topic.  Buff 
and  Campbell  (2002)  define  feedback  as  a  mean  for  the  learner  to  alter  his  understanding 
of  a  task,  by  confirming  adding,  overwriting  tuning  or  reconstructing  previous  knowledge 
basis.  Cohen  (1985,  p.  33)  stated  that  feedback  “is  one  of  the  more  instructionally 
powerful  and  least  understood  features  in  instructional  design.”  Ilgen,  Fisher,  and  Taylor, 
(1979)  concluded  that  feedback  is  a  crucial  piece  in  the  learning  process.  Despite  the 
evidences  that  feedback  is  useful,  there  is  still  a  debate  on  the  kind  of  feedback  or  what 
type  of  information  should  be  delivered  to  better  help  in  the  learning  process.  This  thesis 
explored  several  types  of  feedback  and  associated  with  improved  performance. 

1.  Outcome-based  Feedback 

Outcome-based  feedback  is  based  only  in  the  student’s  achieved  performance  in 
that  training  session.  Normally,  outcome-based  feedback  is  provided  as  percentages  and 
in  messages  such  as:  “You  were  accurate  85%  of  the  time  when  performing  this  task.” 
This  type  of  feedback  is  only  delivered  after  the  training  is  complete.  Numerous  studies 
examining  the  effect  of  outcome-based  feedback  on  trainee  performance  (see:  Todd  and 
Hammond  [1965],  Summers,  Taliaferro  and  Fletcher  [1970],  Hammond  [1971],  and 
Deane,  Hammond,  and  Summers  [1972])  failed  to  find  significant  increase  in 
performance  for  subjects  exposed  to  outcome-based  feedback. 

2.  Process-based  Feedback 

Process-based  feedback  delivers  information  to  the  trainee  about  how  well  they 
performed  on  the  task,  without  informing  ways  to  improve  the  performance  (Kluger  & 
DeNisi,  1996). 

There  are  two  ways  to  present  the  feedback  on  a  process-based  evaluation.  Non¬ 
diagnostic  feedback  does  not  reference  the  trainee’s  performance  but  only  describes  an 
expert  solution  for  that  task.  “A  general  strategy  used  by  an  expert  in  this  field  is...”  or 
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“To  achieve  this  goal,  an  expert  takes  this  or  that  approach...”  are  possible  declarations 
providing  non-diagnostic  process-based  feedback.  Sadler  (1989)  suggests  that  when 
assessing  effective  ways  of  teaching,  feedback  is  only  effective  when  it  addresses  the  gap 
between  what  was  understood  by  the  student  and  the  actual  learning  target. 

Diagnostic  feedback  presents  the  trainee  not  only  the  expert  solution,  but  also  the 
difference  between  what  was  executed  by  the  student  and  the  ideal  outcome.  “Your 
outcome  to  this  task  was  ...  while  the  expert  found  ...,”  “You  took  in  consideration  .... 
aspects  to  your  decision,  while  the  expert  normally  focus  in  ...  aspects.”  are  some 
examples  of  diagnostic  process-based  feedback. 

In  their  2002  investigation.  Buff  and  Campbell  assessed  learning  improvements 
between  sessions  with  subjects  exposed  to  outcome-based  and  the  two  processed-based 
forms  of  feedback  cited  above.  The  feedback  was  presented  in  the  intervals  between 
sessions.  They  found  that  there  was  a  significant  increase  in  performance  when  students 
received  non-diagnostic  feedback,  only  a  marginally  significant  improvement  when  using 
diagnostic  feedback,  and  no  significant  differences  in  performance  when  provided  with 
outcome-based  feedback.  The  control  group  in  their  study  also  had  no  significant  increase 
in  performance  without  receiving  feedback. 

3.  Normative  Feedback 

Normative  is  another  type  of  feedback,  which  provides  individuals  with  a 
comparison  about  their  performance  related  to  all  others  in  their  particular  group. 
(Smither,  Wohlers,  &  London,  1995).  “You  are  in  the  90th  percentile.  10%  of  all  the 
participants  performed  better  than  you”  is  a  typical  example  of  this  type  of  feedback. 

Astwood  and  Van  Buskirk,  Cornejo,  and  Dalton  (2008)  conducted  an  experiment 
comparing  process,  normative,  outcome  and  none  (control)  feedback.  Using  a  modified 
version  of  the  Forward  Observer  Personal  Computer-based  Simulator  (FOPCSIM) 
developed  at  the  Naval  Postgraduate  School,  they  compared  the  performance  of 
participants  randomly  assigned  to  all  four  groups,  side-by-side.  Their  results  suggest  that 
process  feedback  group  was  statistically  more  likely  to  perform  better  than  the  other  three 
groups.  Although  the  study  could  not  find  significant  difference  between  normative  and 
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outcome  feedback,  the  authors  suggested  that  their  findings  might  have  been  attributed  to 
a  potential  limitation  in  the  form  of  the  power  of  the  study. 

4.  Velocity  Feedback 

Velocity  feedback  is  the  comparison  of  trainee’s  performance  with  his  own  prior 
performances  on  the  task.  It  is  considered  a  self-referenced  evaluation.  The  term  velocity 
is  used  as  an  analogy  to  the  “physical  concept  of  change  in  distance  over  time”,  or  the 
trainee’s  changing  state  towards  the  goal  (Kozlowski  et  ah,  2001).  The  advantage  of 
velocity  feedback  is  that  the  trainee  can  rate  his  progress  and  compare  it  to  a  predefined 
goal.  An  example  of  velocity  feedback  statement  could  be:  “You  got  ten  answers  correct; 
this  is  a  25%  improvement  over  your  last  attempt.”  Landsberg,  Van  Buskirk,  and 
Astwood  (2010)  investigated  the  comparison  between  process  feedback,  velocity 
feedback,  and  no  feedback  for  control.  This  study  revealed  that  females  achieved  better 
performance  when  receiving  process  feedback,  while  males  who  received  velocity 
feedback,  outperformed  those  given  process  feedback. 

The  research  cited  above  used  scenarios  where  feedback  was  provided  between 
trials  during  the  training  sessions.  Ilgen,  Fisher,  and  Taylor  (1979)  define  feedback  as 
“information  regarding  the  correctness  of  past  performance.”  How  long  in  the  past? 
Would  it  be  better  to  provide  the  feedback  closer  to  the  action  that  generated  it? 

The  advent  of  game-based  training  systems  and  the  availability  of  powerful 
personal  computers  provide  the  tools  to  investigate  these  questions.  By  using  game 
mechanics  such  as  competitive  scoring,  pop-up  messages,  and  auditory  feedback,  it  is 
possible  to  deliver  feedback  closer  to  the  actions  that  generated  the  outcomes.  It  is  even 
possible  to  provide  this  feedback  immediately  after  the  performance  of  an  action  that 
resulted  in  a  specific  outcome.  The  ability  to  program  systems  capable  of  recording, 
assessing,  and  interacting  with  trainees  based  on  individual  actions  during  a  training 
event  has  been  recognized  as  a  revolutionary  leap  for  training.  As  summarized  by 
Kozlowski  et  al.  (citing  Anderson,  1987),  “Feedback  about  errors  should  be  provided 
immediately  if  the  goal  is  to  develop  rapidly  a  strong  procedural  knowledge  base”  (2001, 
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p.  40)  and  “Feedback  should  be  provided  frequently  to  facilitate  the  development  of  self- 
efficacy  during  training  for  a  complex  task”  (Gist  &  Mitchell,  1992). 
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III.  SYSTEM  DESIGN  PROCESS 


The  instructional  system  design  (ISD)  model  used  to  guide  the  creation  of  STTS 
was  based  on  the  traditional  five  phases  of  analysis,  design,  development, 
implementation,  and  evaluation  (ADDIE)  which  was  developed  at  Florida  State 
University  for  the  U.S.  Army  in  1975  (Branson  et  al.,  1975).  This  chapter  will  cover  the 
analysis  and  design  phases. 

A.  TASK  ANALYSIS 

A  task  analysis  was  performed  during  the  early  stages  of  this  thesis  in  order  to 
define  the  appropriate  type  of  mission  to  be  simulated.  The  task  analysis  was  also  used  to 
determine  which  set  of  mission  sub-tasks  were  feasible  to  be  integrated  into  the  STTS. 
Other  analysis  were  performed  considering  limitations  imposed  by  classification  issues, 
technical  restrictions,  and  time  constraints. 

Submarine  operations  are  complex,  diverse,  and  involve  many  different  types  of 
missions.  Each  mission  relies  on  specific  equipment,  tactics,  techniques,  and  procedures 
(TTP),  and  each  TTP  was  a  potential  area  to  be  trained  in  the  simulation.  Initially,  the 
potential  areas  were  narrowed  to  unclassified  operations.  This  was  mandated  by  the 
desire  to  build  a  simulator  that  SOs  could  use  in  a  wide  variety  of  locations,  and 
including  classified  operations  would  severely  limit  the  areas  where  the  STTS  could  be 
used.  Given  that  most  submarine  operations  are  classified,  this  culled  the  majority  of 
potential  missions. 

Eikewise,  the  subjects  for  the  experiment  were  not  going  to  be  Brazilian  SOs,  so 
this  meant  that  it  was  impossible  to  assume  a  basic  level  of  knowledge  of  the  participants. 
This  further  limited  this  research  to  missions  which  could  be  explained  quickly  and  that  a 
novice  could  achieve  a  basic  level  of  proficiency  within  a  reasonable  amount  of  time. 
Since  many  submarine  operations  are  fairly  complex,  this  again  reduced  the  list  of 
potential  missions. 

Therefore,  an  acceptable  mission  was  one  that  could  be  covered  at  an  unclassified 

level  and  taught  quickly.  Additionally,  the  mission  had  to  consider  that  the  experimental 
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sample  would  come  from  a  population  of  mainly  novices.  With  these  limitations  in  mind, 
the  mission  selected  would  still  need  to  be  meaningful  if  applied  to  actual  SOs.  Finally, 
the  technical  requirements  to  build  such  a  trainer  had  to  be  achievable  within  the  scope  of 
a  thesis  at  the  Naval  Postgraduate  School. 

The  mission  selected  for  task  analysis  was  the  Go-Deep  Range  Exercise 
(GODEX).  A  major  factor  in  the  decision  to  use  GODEX  was  founded  on  the  fact  that  it 
relies  heavily  on  operations  performed  while  a  submarine  is  at  PD.  Tasks  performed  in 
the  GODEX  mission  after  evolving  to  SD  were  not  considered.  The  skills  acquired  by 
training  for  the  GODEX  mission  form  the  basis  for  many  of  the  other  tactical  missions 
that  submariners  perform  throughout  their  careers.  Einally,  simulating  the  view  out  of  the 
periscope  does  not  require  sonar  propagation  models,  most  of  which  are  classified. 

Bastos  (2014)  addressed  similar  task  analysis,  focused  on  the  GODEX  mission. 
Eigure  4  presents  the  overall  hierarchical  task  tree  decomposition  for  the  GODEX 
mission.  His  part-task  trainer  was  developed  for  training  on  the  ARE  task.  This  session 
will  analyze  the  whole  GODEX  mission  with  focus  on  the  contact  observation. 


Eigure  4.  Hierarchical  Task  Tree.  Adapted  from  Bastos  (2014). 
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This  hierarchical,  high-level  representation  of  tasks  allows  a  detailed  sub-tasks 
analysis.  Figure  5  illustrates  the  series  of  decision  points  and  new  subtasks  added  to  the 
analysis.  Figure  6  details  the  hierarchical  tasks  that  compose  the  contact  observation. 


Figure  5.  Summarized  Procedural  Diagram.  Adapted  from  Bastos  (2014). 
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Figure  6.  Hierarchical  Task  Tree  for  Contact  Observation. 


The  results  of  a  detailed  analysis  of  each  sub-task  component  of  GODEX  mission 
are  shown  in  Table  2.  Some  adaptions  were  required  to  this  complete  model  to  address 
the  limitations  related  to  classification  and  lack  of  previous  experience  of  the  population. 
Additionally,  some  subtasks  listed  are  related  to  the  operation  of  the  equipment  itself  and 
they  were  not  represented.  The  tasks  not  simulated  in  STTS  are  marked  in  Table  2  with 
an  asterisk. 
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Table  2.  Detailed  Description  of  Tasks.  Adapted  from  Bastos  (2014). 


Step 

Task 

Task  Description 

Remarks 

1 

Traverse  Area  in 
Safety  /  Stay  as  long 
as  possible  in  the  PD 

This  is  the  overall  goal.  The  PO  must 
traverse  an  area  where  the  transit  of  an 
enemy  task  force  is  expected.  PO  should 
keep  the  submarine  as  long  as  possible 
at  PD,  without  changing  course  or  speed 

PO  should  use  the  periscope  the  little  as 
possible  to  avoid  exposure  and  counter¬ 
detection,  while  constantly  assessing  the 
current  safety  status. 

1.1 

Update  Tactical 
Situation 

PO  performs  the  most  adequate 
technique  to  obtain  information  about 
the  surrounding  tactical  situation. 

The  PO  must  assess  the  source  of 
information  to  be  evaluated  based  on  his 
mental  model  of  the  situation  and  on  the 
time  available  before  the  next 
observation  is  due. 

1.2 

Maneuver 

PO  performs  a  depth  change. 

1.2.1 

Evolve  to  SD 

PO  announces  orally  the  type  of  depth 
change  and  the  previously  defined  SD 
value. 

E.g.,  “Emergency  40  meters” 

1.2.2* 

Evolve  to  PD 

PO  announces  orally  the  type  of  depth 
change  and  the  final  desired  depth  at 
PD. 

E.g.,  “Depth,  15  meters” 

1.1.1* 

Evaluation  of 
Sensors’  Information 

PO  obtains  information  of  the  contacts 
straight  from  sensors’  displays  or  from 
supervisors. 

PO  checks  if  the  information  from 
sensors  is  consistent  with  his  model  of 
the  situation. 

1.1.3* 

Evaluate  data 
processed  by  attack 
team 

PO  checks  the  previous  calculated 
solutions,  obtained  by  the  Attack  Team, 
managed  by  the  Executive  Officer. 

PO  compares  solutions  with  his  own 
mental  models 
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Step 

Task 

Task  Description 

Remarks 

1.1.2 

Data  Collection  with 
Periscope 

PO  obtains  information  of  the  contacts 
performing  a  periscope  observation. 

PO  must  judge  which  type  of  periscope 
observation  is  appropriate  based  on  the 
data  he  needs  to  obtain  and  the  time  he 
has  to  do  it. 

1. 1.2.3 

Contact  Observation 

PO  performs  a  Contact  Observation 

1. 1.2.3. 1 

Pre  Observation 
Action 

Set  of  tasks  performed  to  make  the 
periscope  ready  for  an  observation. 
Comprises  all  actions  taken  from  the 
decision  to  raise  the  periscope  to  when 
the  PO  puts  his  eyes  on  the  eyepieces. 

1. 1.2.3. 1.1 

Check  Platform 
Conditions 

Submarine’s  conditions  to  be  checked 
every  time  before  the  periscope  is  raised 

Each  condition  should  be  checked  “at  a 
glance.”  Depending  on  conditions  like 
the  depth  of  the  keel,  speed  over  ground, 
or  hull  inclination  (trim),  the  AO  may 
decide  to  not  raise  the  periscope  until 
corrections  are  made  on  the  platform 
conditions. 

1. 1.2.3. 1.2 

Communicate 

intentions 

After  deciding  that  all  conditions  allow 
for  periscope  raising,  the  AO  announces 
orally  to  Attack  Team  what  kind  of 
observation  he  will  do. 

E.g.,  “Observation  of  Contact  03” 

1. 1.2.3. 1.3 

Command  periscope 
to  Raise 

PO  commands  the  periscope  he  will  use 
to  be  raised 

E.g.,  “Raise  the  One  (or  Two)” 

After  the  PO  commands  the  periscope  to 
be  raised,  an  auxiliary  supervisor  orally 
announces  the  estimated  bearing  of  the 
target  to  be  observed. 
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Step 

Task 

Task  Description 

Remarks 

1.1.2.3.1.4 

Set  Periseope 
Bearing 

PO  turns  the  periscope  to  the  bearing 
where  he  expects  to  begin  his 
observation. 

Assessing  his  mental  model  and  the 
information  just  received,  the  PO 
decides  what  is  the  most  probable 
target’s  bearing.  As  periscope  comes 
closer  to  the  floor  limit,  the  PO 
crouches.  As  the  periscope  handles 
become  reachable  near  the  floor,  the  PO 
opens  them  and  turns  the  periscope  to 
the  chosen  bearing. 

1. 1.2.3. 1.5* 

Check/Set  Periseope 
Controls 

PO  manually  sets  the  periscope  controls 
while  it  is  being  raised 

To  save  time,  the  PO  must  crouch  and 
do  these  settings  as  soon  as  he  can  reach 
the  periscope  controls  near  the  floor, 
while  he  still  cannot  reach  the  eyepieces 
with  his  eyes.  For  Contact  Observation 
PO  sets  /  checks  the  height  of  target’s 
point  to  be  observed  on  the  periscope’s 
front  panel  and  checks  the  zoom  level  is 
set  to  lower. 

1.1.2.3.2 

Observation 

PO  performs  the  chosen  type  of 
observation 

1.1.2.3.2.1 

Set  Periscope  Height 

PO  sets  the  periscope  height 

PO  evaluates,  according  to  his  own 
height  and  sea  state,  a  height  for  the 
periscope  in  which  it  is  possible  to 
execute  the  observation,  exposing  just 
the  sufficient  amount  of  mast. 
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Step 

Task 

Task  Description 

Remarks 

1.1.2.3.2.2 

Set  Periseope  Lens 
Elevation 

PO  uses  periscope  controls  to  set  the 
best  lens  elevation  to  execute  the 
observation. 

PO  will  execute  this  throughout  the 
observation  looking  for  keep  the  target 
in  the  center  of  periscope  screen, 
compensating  inclinations  caused  by  the 
sea  state. 

1.1.2.3.2.3 

Set  Zoom  To  High 

PO  sets  the  periscope  zoom  to  high. 

Only  set  zoom  to  high  if  the  target  to  be 
observed  is  close  to  the  center  of  the 
periscope  screen 

1.1.2.3.2.4 

Set  Periseope  to 
Target’s  Bearing  / 
Mark  Bearing 

PO  rotates  the  periscope  to  align  the 
vertical  line  in  the  screen  to  the  center  of 
the  target,  press  the  mark  bearing  button 
to  send  the  current  bearing  to  the  combat 
system,  and  disseminate  orally  the 
bearing  reading. 

E.g.,  “Bearing  040” 

PO  should  be  careful  to  always  take  the 
bearing  form  the  same  point  of  the 
target. 

1.1.2.3.2.5 

Set  Stadimeter  to 
Target’s  Height  / 
Mark  Distanee 

PO  rotates  the  stadimeter’ s  wheel  to  the 
point  where  the  reference  point  of  the 
target  is  aligned  to  the  ship’s  water  line, 
press  the  mark  distance  button  to  send 
the  current  range  reading  to  the  combat 
system,  and  disseminate  orally  the  range 
reading. 

E.g.,  “Distance  3500  yards” 

PO  should  compensate  for  distant 
contacts  (more  than  4  kyds)  where  the 
ship’s  water  line  cannot  be  seen,  due  to 
earth’s  curvature. 

1.1.2.3.3* 

Post  Observation 
Actions 

PO  takes  actions  to  prepare  the 
periscope  for  the  next  observation 
before  lowering  it. 

According  to  the  current  tactical 
situation,  the  PO 

must  predict  what  is  his  next  step,  and 
decide  to  lower  the  periscope  all  the  way 
down  or  just  enough  to  break  its 
exposition  (“flood”)  allowing  for  a  faster 
raising  afterwards. 
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Step 

Task 

Task  Description 

Remarks 

1.1.2.3.3.1 

Set  Stadimeter  to 
Zero 

PO  rotates  the  stadimeter’ s  wheel  all  the 
way  back  to  the  point  where  the  images 
of  both  eyepieces  are  aligned. 

Range  reading  shows  16000  yds. 

1.1.2.3.3.2 

Set  Zoom  to  Low 

PO  sets  the  zoom  back  to  low. 

1.1.2.3.3.3 

Command  Periseope 
to  Lower 

PO  commands  orally  to  lower  or  “ 
flood”  the  periscope 

E.g.,  “Lower  the  One  (or  Two)”  or 
“Flood” 

1.1.2.3.3.4 

Reposition  Periseope 

While  the  periscope  is  being  lowered, 
the  PO  quickly  turns  it  to  the  start 
bearing  of  the  next  observation 

PO  must  plan  one  step  ahead,  choosing 
the  next  observation  to  be  done,  and  set 
the  periscope  bearing  to  the  direction  of 
the  next  observation.  This  movement 
must  be  done  quickly,  without  looking  at 
the  eyepieces. 

1.1.2.3.3.5* 

Close  Handles 

PO  close  periscope  handles. 

1.1.2.3.4 

Report  Results 

PO  recall  from  memory  the  data  about 
the  observation,  calculates  and  orally 
reports  the  results 

1.1.2.3.4.1 

Report  Angle-on-the- 
bow  Side 

PO  orally  reports  the  side  of  the  target 
that  he  saw  during  the  observation. 

If  the  target  is  pointing  directly  to  the 
submarine,  no  side  should  be  informed. 

1.1.2.3.4.2 

Report  Angle-on-the- 
bow  Value 

PO  orally  reports  the  angle  from  the 
target’s  bow  to  the  bearing  where  the 
submarine  is. 

E.g.,  “Angle-on-the-bow  Starboard  40” 

1.1.2.3.4.3* 

Report  Look  Interval 

PO  mentally  calculates  the  LI,  based  on 
the  observed  target’s  range. 

E.g.,  “Eook  Interval  2  minutes  and  30 
seconds” 
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Figure  7  illustrates  the  final  proeedural  diagram  for  STTS,  after  all  the 
eonsiderations  listed  previously. 


Figure  7.  STTS  Procedural  Diagram. 


B.  LEARNING  OBJECTIVES 

After  a  training  section  on  STTS,  a  participant  should  be  able  to: 

•  Enumerate  the  correct  sequence  of  movements  for  each  contact  observation. 

•  List  the  pre-observation  required  actions. 

•  Describe  the  process  of  taking  a  contact’s  bearing. 

•  Describe  the  process  of  taking  a  contact’s  distance; 

•  Enumerate  the  actions  to  prepare  the  periscope  for  the  next  observation, 
before  lowering  it. 
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•  Explain  what  the  AOB  is  and  how  it  is  disseminated. 

•  Report  the  criteria  for  a  correct  emergency  call. 

These  learning  objectives  reflect  only  the  set  of  KSA  intended  to  be  assimilated 
by  a  participant  on  this  study.  Of  all  knowledge  required  for  the  GODEX  mission,  this 
simulation  addresses  only  the  basic  unclassified  process. 

C.  SYSTEM  REQUIREMENTS 

In  order  to  achieve  the  desired  learning  objectives,  STTS  should  be  able  to  meet 
the  following  requirements: 

1 .  Simulate  the  virtual  environment  outside  the  submarine 

a.  Sea  surface  and  its  movements 

b.  Sky 

2.  Simulate  the  virtual  environment  inside  the  submarine  (control  room) 

a.  Periscope  3D  Model  representation 

b.  Periscope  rotation  while  not  completely  raise 

c.  Depth  indication 

d.  Trim  indication 

3.  Simulate  the  visual  through  the  periscope  screen 

a.  Built-in  true  bearing  indicator 

b.  Built-in  relative  bearing  indicator 

c.  Built-in  stadimetric  range  indicator 

4.  Simulate  the  periscope  equipment 

a.  Horizontal  rotation  with  at  least  two  speeds  (fine  and  coarse) 

b.  Vertical  rotation  of  the  lenses 

c.  Vertical  raise  and  lower  movements 
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d.  Stadimeter 


e.  Zoom  levels 

5.  Simulate  enemy  surface  ships  and  their  movements 

6.  Simulate  orders  and  reports  made  by  PO 

7.  Organize  experiment  by  three  trials 

8.  Score  management 

9.  Automatically  assign  participants  to  groups  keeping  balanced  experience 
scores 

10.  Provide  feedback  according  to  assigned  group 
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IV.  SYSTEM  DEVELOPMENT 


A.  OVERVIEW 

The  intent  of  this  thesis  was  to  build  a  realistic  simulation  for  submarine  officer 
tactical  training  and  a  suitable  platform  for  testing  the  hypothesis.  Contact  Observation 
was  the  task  selected  during  the  task  analysis,  based  mainly  on  unclassified  part-tasks  that 
could  be  taught  in  a  short  window  of  time  for  non- submariners.  However,  additional 
factors  steered  the  system  development  towards  distinct  goals.  The  main  constraint  on  the 
development  of  a  tactically  realistic  system  lies  on  classification  of  some  concepts  of  the 
task.  It  was  also  necessary  to  adapt  and  simplify  the  execution  of  the  task.  Some 
adaptations  had  to  be  made  to  the  actual  procedure  so  that  participants  with  no  previous 
knowledge  would  be  able  to  accomplish  the  GODEX  task.  One  example  is  the  calculation 
of  look  intervals:  in  reality,  this  calculation  must  accomplished  mentally  in  a  very  short 
timeframe  after  lowering  the  periscope,  considering  the  distance  observed  for  the  contact, 
the  maximum  speed  of  a  particular  type  of  ship,  and  the  go-deep  range  for  the  bearing 
sector  where  the  contact  is  located.  On  this  version  of  STTS,  the  LI  is  calculated 
automatically  and  a  stopwatch  correspondent  to  that  contact  is  automatically  updated  to 
reflect  the  LI.  This  chapter  describes  the  process  used  for  developing  a  version  of  STTS 
that  is  flexible  for  use  as  an  unclassified  experimental  testbed  as  well  as  a  basis  for  the 
future  development  of  a  more  tactically  realistic  training  tool. 

B.  SOFTWARE  USED 

1.  Unity 

Unity  3D  is  a  popular  game  engine  developed  by  Unity  Technologies  from 
Denmark.  It  is  a  cross-platform  solution,  which  allows  developers  to  build  their  games  in 
one  development  environment  and  release  them  for  multiple  platforms,  such  as  different 
consoles,  numerous  operating  systems,  smartphones,  and  web-browsers  (“Unity,”  n.d.). 

Numerous  reasons  guided  the  selection  of  Unity  for  the  development  of  the  STTS. 
The  first  is  that  Unity  allows  scripting  in  either  C#  or  UnityScript  (a  language  based  on 

JavaScript).  C#  is  very  similar  to  JAVA  programming  language,  which  the  author  has 
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significant  previous  experience,  allowing  an  easy  transition.  The  second  reason  is  that  the 
popularity  of  this  game  engine  has  produced  a  large  and  collaborative  developer 
community,  which  uses  a  several  forums  and  blogs  that  facilitate  the  process  of  learning 
Unity  for  new  developers.  Similarly,  online  documentation  is  complete,  easily  accessible, 
and  includes  implementation  examples.  Additionally,  the  Unity  Asset  Store  is  an  online 
repository  where  any  developer  can  publish  or  sell  their  solutions.  For  STTS 
development,  external  assets  were  used  to  build  the  ocean  model,  the  skybox,  and  the 
graphical  user  interface  (GUI).  Lastly,  it  is  possible  to  release  final  applications  for  many 
platforms,  operating  systems  and  consoles,  with  none  or  very  few  adaptations  needed  to 
the  main  source  code.  The  final  STTS  application  was  built  for  Windows  personal 
computers  but  Unity  allows  easy  conversion  to  Mac  OS,  Android,  iOS,  and  other 
operating  systems,  which  might  be  useful  for  future  iterations  of  the  STTS. 

2.  Blender 

Blender  is  a  multi-platform,  open-source  3D  tool  for  modeling,  animations  and 
game  development.  In  spite  of  being  open-source,  it  can  be  compared  to  other  proprietary 
and  expensive  competitors  in  terms  of  creation  power.  From  individual  developers  to  big 
companies.  Blender  has  a  large  adoption  worldwide,  which  contributes  for  collaboration 
and  learning  process.  The  3D  environment  for  the  submarine  control  room  and  the 
periscope  model  were  built  using  Blender  (“About,”  2016). 

3.  GIMP 

GNU  Image  Manipulation  Program  (GIMP)  is  a  powerful  open-source  tool  for 
image  manipulation.  It  was  used  to  create  and  edit  the  textures  used  in  the  submarine 
model  and  the  periscope  screen  mask. 

4.  Microsoft  Visual  Studio 

Visual  Studio  is  the  integrated  development  environment  (IDE)  chosen  to  code 
for  Unity  game  engine  because  it  has  C#  as  one  of  the  supported  languages.  Visual  Studio 
Tools  for  Unity  is  a  plug-in  that  allows  the  visualization  of  scripts  following  the  same 
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hierarchy  found  in  Unity.  It  also  provides  a  Unity  console  inside  Visual  Studio,  which  is 
a  very  useful  tool  for  debugging. 


C.  MODELS 

This  section  lists  and  describes  the  models  used  to  build  STTS  and  the  technical 
approaches  to  integrate  them. 

1.  External  Environment 

a.  Ocean 

The  ocean  model  used  is  a  part  of  a  project  created  by  an  individual  developer 
(Hawkins,  2013).  This  asset  was  retrieved  from  his  blog,  where  is  possible  to  find  all  the 
documentation  and  the  files  to  download.  This  asset  is  based  on  a  radial  grid  created  over 
a  plane  and  the  vertices  of  this  grid  are  visually  moved  up  and  down  to  create  the  wave 
effect.  For  performance  purposes,  the  heavy  fast  Fourier  transform  (FFT)  calculations  for 
each  individual  vertex  of  the  grid  are  done  on  the  GPU  and  fetched  once  per  frame.  Since 
the  data  is  calculated  on  the  GPU,  the  vertices  are  not  physically  moved.  The  appearance 
of  movement  is  created  by  the  height  maps  generated  from  the  FFT  done  on  the  GPU. 
This  asset  allows  modifying  the  seas  by  controlling  wind  speed,  wave  period,  and 
amplitude.  It  uses  a  light  source  as  reference  for  realistic  sun  reflections.  The  lack  of  real 
geometry  being  moved  in  the  scene  grants  the  model  an  optimized  performance,  however 
it  does  not  allow  the  use  of  geometry  for  physics  calculations,  such  as  buoyancy. 

To  address  the  issue  of  lack  of  buoyancy  movements  in  the  vessels,  a  script  was 
implemented  to  rotate  the  submarine  and  the  targets  around  the  original  “heads-up” 
position,  using  Perlin  noise.  Originally  created  to  generate  naturalistic  procedural  textures 
(Perlin,  2002),  the  Perlin  noise  was  adapted  to  generate  smooth  continuously  changing 
movement  along  both  longitudinal  (roll)  and  horizontal  (pitch)  ships’  axes.  The  Perlin 
noise  was  applied  over  a  long  amplitude  sine  wave  and  adjusted  using  specific  factors  for 
roll  and  pitch  (see  Figure  8). 
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//Noise  to  simulate  waves: 

float  noise  =  Mathf .Sin(Time.realtimeSinceStartup  *  waveFreq)  * 
Mathf.PerlinNoise(Time.realtimeSinceStartupj0f)  /  waveFreq; 

float  noisePitch  =  noise  *  waveMagnitude  *  pitchFactor; 
float  noiseRoll  =  noise  *  waveMagnitude  *  rollFactor; 

//Rotation  to  simulate  waves: 

transform. Rotate(new  Vector3(noisePitchj  0,  noiseRoll)); 


Figure  8.  Perlin  Noise  Procedural  Wave  Movements. 


b.  Skybox 

The  environmental  background  was  created  using  a  skybox  downloaded  together 
with  the  ocean  model  (see  Figure  9). 


Figure  9.  Periscope  View  of  the  Ocean,  Skybox,  and  Target. 
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c. 


Earth  Curvature 


Due  to  the  curvature  of  the  earth,  at  great  distances  the  apparent  waterline  of  a 
ship  when  viewed  through  a  periscope  differs  from  the  actual  waterline.  Thus,  at  great 
ranges,  it  affects  the  distance  reported  by  the  periscope’s  stadimeter,  and  the  PO  must 
consider  this  effect  when  determining  distance.  Therefore,  the  game  was  designed  to 
reflect  this  to  mitigate  the  potential  for  negative  training. 

To  account  for  this,  a  simple  mathematical  model  was  implemented  to  simulate 
the  earth’s  curvature  effect  (ECE)  on  distant  objects,  which  appear  to  be  as  “deeper”  in 
the  water  than  they  are.  Eigure  10  shows  a  schematic  diagram  and  the  variables  involved. 
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Eigure  10.  Earth’s  Curvature  Effect. 


Eirst,  the  distance  to  horizon  for  the  submarine  was  calculated  (dl)  based  on  the 
periscope  height  (h).  The  Pythagorean  Theorem  was  used  on  the  triangle  CHP,  since  the 
line  from  the  periscope  lens  to  the  target  (segment  P-T)  is  tangent  to  the  earth  on  the 
horizon  (point  H). 


(r  +  hf=r^+d,^ 

+  2rh  +  h^  =r^  +  d} 


(3) 


Considering  the  Earth  radius  r  =  6,967,410  yards  and  the  mean  periscope 
height /i  =  1.0  yards  above  the  sea  level,  the  virtual  horizon  is  dl  =  3732  yards. 
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To  calculate  dynamically  the  ECE  (e),  the  same  process  was  used  for  the  triangle 


CHT. 


(r  +  ef  =r"+d/ 

+  2re  +  e^  =r^  + 

=  2re  + 

Once  e  is  significantly  smaller  then  2re  the  simplified  formula  for  e  is: 


2r 


Also  considering: 


(4) 


(5) 


d^  =  Range -3132 


(6) 


Substituting  (7)  on  (6),  we  have  the  final  formula  for  ECE  given  by  Equation  (8): 


e  = 


{Range  -  3132) 
2r 


(7) 


This  effect  is  applied  to  the  simulation  (if  the  range  is  greater  than  3732  yards)  by 
subtracting  e  from  the  “y”  coordinate  of  the  targets,  once  per  frame,  causing  them  to  be 
deeper  in  the  water  plane  proportional  to  the  distance  to  the  submarine. 


2.  Ships 

The  3D  models  of  frigate  type  22  (Brazilian  Navy  Class  Greenhalgh)  were  kindly 
provided  from  Brazilian  Navy  Modeling  and  Simulation  Office.  Originally  developed  in 
Autodesk  3DS  Max,  the  model  used  an  .fpx  extension.  The  model  can  be  directly 
imported  into  Unity,  which  removed  the  need  for  additional  processing.  A  comparison  of 
the  model  and  a  picture  of  the  actual  frigate  is  shown  in  Eigure  1 1 . 
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Figure  11.  Frigate  Greenhalgh  (Type  22)  Model  Loaded  in  Unity  (above)  and 
Real  Picture  (below).  Source:  Area  Militar  (2011). 

3.  Submarine 

The  only  visible  part  of  the  submarine  in  the  STTS  is  the  control  room’s  interior. 
Requirements  for  a  detailed  model  of  the  control  room  did  not  emerge  from  the  task 
analysis  so  it  is  presented  as  a  rough  approximation.  The  main  factor  for  including  a  3D 
environment  for  the  control  room  was  the  need  to  simulate  the  rotation  of  the  periscope 
while  it  goes  up  or  down  (steps  1.1. 2. 3. 3.4  and  1. 1.2.3. 1.4  on  the  task  analysis).  The  3D 
model  was  considered  important  to  help  the  participants  to  position  themselves  spatially 
when  rotating  the  periscope  during  ascent  and  descent.  In  this  scene,  it  is  possible  to  see 
the  three  metallic  tubes,  which  compose  the  periscope:  two  lateral  hydraulic  pistons  and, 
in  the  center,  the  periscope  itself.  Simple  cubes  were  used  to  represent  the  walls  forming 
the  forward  compartment.  A  cylinder  represents  the  submarine’s  hull.  Three  fully 
functional  indicators  are  positioned  on  the  walls  and  include  a  wall  clock  on  the  right,  a 
digital  depth  indicator  showing  the  keel  depth  in  meters,  and  a  “bubble”  indicator, 
showing  the  hull  inclination  on  the  left  side  (necessary  to  accomplish  step  1.1. 2. 3. 1.1  on 
the  task  analysis).  To  provide  information  required  to  complete  the  simulated  GODEX 
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task,  three  stopwatches  are  displayed  on  the  bottom  of  the  viewport.  This  mimics  the 
stopwatches  used,  in  a  real  GODEX  mission  to  maintain  Lis.  Figure  12  illustrates  the 
interior  view. 


Figure  12.  Control  Room  Simulated  in  STTS. 


The  inside  and  outside  of  the  submarine  are  two  distinct  game  objects  in  the 
scene,  as  shown  in  Figure  13.  The  only  connection  between  the  two  are  the  movements 
applied  to  the  physical  periscope  that  holds  the  cameras  on  the  outside  submarine  and  the 
representation  of  the  periscope  inside  submarine.  Figure  13  shows  how  the  objects  are 
connected  in  Unity. 
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Figure  13.  Unity  Game  Objeets  Hierarchy  Showing  Two  Distinct  Game  Objects 

for  Interior  and  Exterior  Submarine  Models. 


4.  Periscope  Display 

Once  the  periscope  is  raised  the  viewport  changes  to  the  outside  view,  simulating 
the  image  perceived  by  the  PO  while  looking  through  the  periscope  eyepieces. 

a.  Periscope  Mask 

A  peripheral  blur  is  a  visual  artifact  experienced  when  SOs  use  periscopes.  In 
order  to  simulate  this  effect,  a  texture  mask  was  used  over  the  periscope  camera  in  the 
STTS.  This  texture  was  created  by  applying  consecutive  growing  transparent  circles  to  a 
black  background  image.  This  process  resulted  in  a  simulated  blur  observed  on  the  view’s 
peripheries  when  looking  through  the  periscope.  Figure  14  shows  this  effect.  Masking 
was  also  used  to  model  the  horizontal  and  vertical  reticle  patterns.  The  upper  digital 
display  indicates  the  range  when  the  stadimeter  is  applied.  The  lower  left  display 
indicates  the  true  bearing  and  the  lower  right  display  indicates  the  relative  bearing.  In  this 
scenario,  the  bearing  indications  are  always  the  same  because  the  submarine  course  is 
fixed  as  000°  (North). 
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Figure  14.  Outside  View  with  Stadimeter  Applied. 


b.  Stadimeter 

The  stadimeter  is  a  very  important  instrument  onboard  a  submarine  and  it  is  used 
by  SOs  when  finding  the  range  of  contacts.  The  process  of  using  a  stadimeter  to 
determine  range  consists  of  employing  trigonometric  determination  of  one  side  of  a  right 
triangle,  by  knowing  one  angle  and  the  length  of  the  opposite  side.  By  rotating  the  range 
control  knob  in  the  periscope,  an  angle  (0)  is  applied  to  one  of  the  lenses  creating  the 
lagging  effect  visualized  in  Figure  14.  The  known  side  of  the  right  triangle  is  the  target’s 
height.  This  value  in  feet  must  be  pre-set  in  the  periscope.  To  have  a  correct  measurement 
of  target’s  range,  the  periscope  operator  must  align  lower  edge  of  the  upper  image  to  the 
upper  edge  of  the  lower  image.  For  this  simulation,  the  upper  and  lower  references  are 
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the  rear  yardarm  (horizontal  mast)  and  the  ship’s  waterline,  respectively.  Figure  15  and 
Equation  (9)  show  the  triangle  and  the  formula  used  to  calculate  the  target’s  range. 


is£= _ ^ 


Figure  15.  Stadimeter’s  Operation  Principle. 


tan6> 


(8) 


This  operation  principle  was  simulated  in  STTS  by  the  use  of  two  cameras 
positioned  in  the  exact  same  place  on  the  top  of  periscope  mast.  By  rotating  the  mouse 
wheel  down,  a  proportional  angle  up  (0)  is  applied  to  one  of  the  cameras,  resulting  in  the 
image  in  this  camera  being  translated  down.  Every  frame,  the  image  captured  by  this 
camera  is  converted  into  a  raw  image  and  the  transparency  necessary  for  the  desired 
effect  is  applied.  This  treated  raw  image  is  projected  in  a  GUI  layer,  in  front  of  the 
original  camera,  and  rendered  together  with  the  rest  of  the  scene.  When  angle  0  is  close 
to  zero,  the  stadimeter  camera  is  disabled  and  there  is  no  overlay  to  optimize 
performance. 

c.  Zoom  Levels 

The  zoom  levels  simulated  in  STTS  are  1.5x  (low  zoom)  and  6x  (high  zoom).  To 
implement  it  in  the  simulation,  the  field-of-view  (FOV)  camera  property  was  adjusted  in 
both  main  and  stadimeter  cameras.  In  low  zoom,  the  cameras  are  set  to  32  degrees  FOV 
and  in  high  zoom  they  are  set  to  8  degrees  FOV. 
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d.  Periscope  Controls 

Table  3  provides  a  list  of  all  mouse  and  keyboard  commands  used  to  control  the 
periscope. 


Table  3.  Periscope  Controls  List. 


ACTION 

MOUSE 

KEYBOARD 

Rotate  left 

Feft  mouse  button 

Feft  arrow  or  “D” 

Rotate  right 

Right  mouse  button 

Right  arrow  or  “A” 

Rotate  slower 

- 

-1-  Shifta 

Rotate  faster 

- 

-1-  Controla 

Adjust  lens  up 

- 

Up  Arrow  or  “W” 

Adjust  lens  down 

- 

Down  Arrow  or  “S” 

Adjust  height  up 

- 

“Q” 

Adjust  height  down 

- 

“E” 

Swap  zoom 

Middle  mouse  button 

“Z” 

Stadimeter  increase  0 

Rotate  mouse  wheel  down 

- 

Stadimeter  decrease  0 

Rotate  mouse  wheel  up 

- 

“  Keys  to  be  pressed  together  with  regular  rotation  command. 


5.  GUI 

The  graphical  user  interface  (GUI)  is  a  very  important  component  of  this 
simulation.  It  was  designed  to  be  able  to  simulate  the  dynamic  actions  of  a  real  exercise 
on  board  a  submarine.  Each  button  pressed  corresponds  to  a  command  or  report  made  in 
real  life.  The  sequence  of  commands  and  reports  is  also  an  important  learning  objective, 
so  the  participant  must  follow  the  top  down  sequence  in  the  simulation  or  a  penalty  score 
would  be  applied.  The  buttons’  colors  are  representative  of  their  states,  being  red  for 
available  and  green  for  already  pressed  buttons  (see  Figure  16).  The  “Emergency  40 
meters”  button  is  always  available.  Only  mouse  clicks  and  mouse-wheel  rotations  are 
used  as  input  methods.  To  inform  the  AOB  value,  the  user  rotates  the  mouse- wheel  to  set 
the  estimated  angle  and  finish  the  observations  by  clicking  the  “Submit”  button. 

The  upper  part  of  the  right  panel  on  the  GUI  is  used  for  display  both  raw  and 
adjusted  score.  For  participants  on  the  RTF  group,  the  lower  part  displays  a  log  of 
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previous  messages  allowing  them  to  review  their  actions,  while  waiting  to  raise  the 
periscope.  For  participants  on  PHF  group,  this  message  log  is  disabled. 


Figure  16.  Graphical  User  Interface  for  RTF  Group. 


D.  SCENARIO  DEVELOPMENT 

Based  on  the  LOs,  the  scenario  was  designed  to  facilitate  the  comprehension  of 
the  concepts  taught.  It  was  also  considered  the  desired  flow,  in  order  to  maximize  the 
number  of  contact  observation  executed  in  each  trial.  Three  surface  contacts  were 
positioned,  such  that  the  participant  could  have  enough  time  to  observe  them  as  they 
approach,  following  the  PDOT.  The  participants  were  informed  that  the  GDR  considered 
for  this  scenario  was  1000  yards.  If  any  contact  gets  closer  than  1200  yards,  it  is 
considered  a  good  time  to  maneuver  to  SD. 

Each  run  was  designed  to  take  between  seven  minutes  to  exactly  nine  minutes  and 
24  seconds  to  be  completed  depending  on  the  moment  that  the  decision  to  dive  is  made. 
The  correct  timeframe  for  diving  occurred  from  eight  minutes  and  36  seconds  to  nine 
minutes.  If  the  participant  failed  to  abandon  the  PD,  the  run  is  finished  when  a  contact 
reaches  900  yards. 
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E.  AUTOMATIC  GROUP  ASSIGNMENT 


The  algorithm  detailed  in  Appendix  A  was  used  to  automatically  assign 
participants  to  either  PHF  or  RTF  group.  This  approach  enabled  each  experimental  group 
to  have  equivalent  distribution  of  experience  scores.  The  experience  score  is  based  on 
answers  to  “YES”  or  “NO”  questions  on  the  demographics  survey. 

If  an  experimental  group  had  four  more  subjects  than  the  other,  then  the  next 
subject  is  automatically  assigned  to  the  group  with  fewer  subjects.  Otherwise,  the  mean 
experience  score  in  each  group  is  established.  The  new  mean  scores  are  calculated  for 
each  group  considering  the  inclusion  of  the  new  participant  in  that  group. 

After  comparing  new  and  old  mean  scores  the  new  participant,  is  added  to  the 
group  that  would  generate  the  smaller  difference  of  experience  scores.  If  the  difference 
between  scores  is  the  same  regardless  of  which  group  the  participant  is  assigned  to,  the 
algorithm  chooses  the  group  with  lower  number  of  participants. 

The  actual  assignment  occurs  by  setting  the  Boolean  variable  hasFeedback  to 
“true”  if  RTF  is  the  chosen  group  or  “false”  otherwise.  A  new  line  is  created  on  the  setup 
file  correspondent  to  the  chosen  group  to  include  the  participant’s  data.  The  final  mean 
experience  score  was  17.35  for  PHF  group  and  17.05  for  RTF  group,  with  17  participants 
in  each. 

F.  SCORE  MANAGEMENT 

In  order  to  evaluate  performance,  each  subject  received  a  score  for  each  run  in 
STTS.  These  scores  were  calculated  by  assigning  each  learning  objective  a  score  defined 
for  partial  or  complete  achievement  of  that  part-task.  Some  tasks  were  binary:  either  the 
subject  performed  them  or  not,  such  as  “Chose  Correct  Contact.”  For  these,  subjects 
either  gained  or  lost  points.  Other  tasks  could  be  performed  to  varying  degrees  as 
competency,  such  as  correctly  estimating  the  range  or  bearing  to  a  contact.  For  these, 
subjects’  performance  was  ranked  “Excellent,”  “Good,”  “Average”  or  “Poor,”  and 
subjects  received  or  lost  points  based  upon  which  category  their  performance  fell  into. 
Table  4  shows  the  tasks  and  their  associated  points. 
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Table  4.  Raw  Score  Components. 


ACTION 

SCORE 

Choose  wrong  contact 

-100 

Choose  correct  contact 

-tlOO 

Raise  periscope  with  less  than  20  seconds  remaining  in  the  Stopwatch 

-tlOO 

Bearing  mark  precision: 

Bearing  error  <0.1  degrees  -  EXCELLENT 

-t200 

0.1  degrees  <  Bearing  error  <  0.2  degrees  -  GOOD 

-tlOO 

0.2  degrees  <  Bearing  error  <  0.4  degrees  -  AVERAGE 

-t50 

Bearing  error  >  0.4  degrees  -  POOR 

-100 

Distance  mark  precision 

Distance  Percentage  error  <  5%  -  EXCELLENT 

-t200 

5%  <  Distance  Percentage  Error  <  10%  -  GOOD 

-tlOO 

10%  <  Distance  Percentage  Error  <  15%  -  AVERAGE 

-t50 

Distance  Percentage  Error  >  15%  -  POOR 

-100 

Lower  with  stadimeter  not  zeroed  and  zoom  applied 

-200 

Lower  with  stadimeter  not  zeroed 

-100 

Lower  with  zoom  applied 

-100 

Lower  without  zoom  or  stadimeter  applied 

-tlOO 

Choose  wrong  AOB  side 

-100 

AOB  estimation  precision: 

AOB  error  <  3  degrees  -  EXCELLENT 

-t200 

3  degrees  <  AOB  error  <  6  degrees  -  GOOD 

-tlOO 

6  degrees  <  AOB  Error  <12  degrees  -  AVERAGE 

-t50 

AOB  error  >12  degrees  -  POOR 

-100 

Emergency  call  scoring  criteria: 

Emergency  Distance  >  1600 

-400 

1600  >  Emergency  Distance  >  1500 

-200 

1500>  Emergency  Distance  >  1300 

-100 

1200  >  Emergency  Distance  >  1000 

-t400 

1000  >  Emergency  Distance  >  900 

-200 

Emergency  Distance  <  900 

-400 

Button  clicked  out  of  sequence 

-100 

Stopwatch  reached  zero 

-200 

Periscope  raised  for  more  than  35  seconds 

-100 

Contact  range  reached  1000  yards 

-200 

After  each  run,  points  for  each  subject’s  actions  were  summed  to  create  a  raw 
score,  which  reflected  how  well  the  subject  carried  out  all  the  given  tasks  but  one.  The 
other  task  was  minimizing  the  percentage  of  time  that  the  periscope  was  up  during  the 
trial,  known  in  the  submarine  community  as  submarine  indiscretion  ratio  (SIR).  It  was 
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continuously  calculated  by  dividing  the  accumulated  time  between  “Raise  Periscope”  and 
“Lower  Periscope”  actions  by  the  total  time  in  the  run.  To  penalize  subjects  for  a  high 
SIR,  an  adjusted  score  was  calculated  for  each  run. 

The  adjusted  score  was  calculated  based  on  the  raw  score  and  the  SIR.  The 
optimal  ratio  between  precision  and  speed  factors  were  defined  based  on  preliminary 
experimental  tests  with  STTS.  Without  including  a  penalty,  a  participant  could  increase 
his/her  raw  score  by  making  more  observations,  which  was  tactically  incorrect  because  it 
increased  periscope  exposure.  Including  a  penalty  for  the  amount  of  time  the  periscope 
was  raised  would  result  in  a  decreased  adjusted  score.  Experimentation  based  upon  the 
designer’s  experience  as  a  submarine  officer  found  that  by  dividing  raw  score  by  SIR 
cubed  was  an  appropriate  penalty.  The  constant  20  was  use  to  keep  the  adjusted  score  in  a 
reasonable  range  around  the  raw  score.  Equation  10  shows  the  formula  for  adjusted  score 
calculation. 


AdjScore 


RawScore 
SIR^  X  20 


(9) 


G.  DATA  COLLECTION 

All  data  generated  during  the  training  sessions  was  stored  as  tables  in  .csv  files. 
The  system  generates  one  file  per  run  for  all  participants,  and  stores  each  observation  as  a 
line  in  the  file.  In  that  lines  the  system  records  all  possible  errors  committed  during  the 
observation  as  Boolean  variables  (true  if  the  error  existed  or  false  otherwise),  numeric 
precision  errors  (like  bearing  error,  AOB  estimation  error,  etc.),  raw  score,  SIR,  adjusted 
score  and  times  spent  in  each  action.  The  primary  variables  of  interest  for  the  experiment 
are  raw  score,  SIR  and  adjusted  score.  The  others  were  recorded  in  case  additional 
questions  needed  to  be  answered  during  data  analysis. 

The  simulation  also  generates  one  file  per  trial,  per  participant  to  store  the 
feedback  generated  during  the  run.  These  files  were  presented  to  the  PHE  group 
participants  in  the  end  of  each  trial. 
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V.  EXPERIMENT 


A.  DESIGN  OF  EXPERIMENT 

In  order  to  answer  research  the  research  questions,  this  experiment  was  designed 
to  determine  if  different  types  of  feedback  (RTF  and  PHF)  affected  participants’ 
performance.  Exploratory  investigations  examined  if  various  demographic  factors  (age, 
time  in  service,  military  affiliation,  and  gaming  experience)  affected  the  efficacy  of  these 
types  of  feedback.  Participants  were  assigned  to  one  of  two  treatment  groups  (PHF  or 
RTF)  which  were  balanced  by  the  subjects’  experience  levels.  The  factors  used  to  define 
this  experience  were  asked  as  “YES”  or  “NO”  questions,  as  described  in  section  IV.E. 
Each  group  performed  two  runs  in  the  STTS,  which  varied  in  the  type  of  feedback:  one 
group  received  real-time  feedback,  while  the  other  received  post-hoc  feedback  after 
completion  of  the  run.  After  these  two  runs,  each  group  performed  a  final  run  with  no 
feedback  delivered.  All  three  runs  were  scored  using  an  algorithm  described  in  the 
section  IV.E. 

B.  RECRUITMENT 

Following  the  approved  protocol  (Appendix  B),  subjects  were  recruited  by 
posting  copies  of  the  recruitment  poster  (Appendix  C)  across  campus  and  by  sending  a 
bulk  email  to  all  NPS  students  (Appendix  D). 

C.  PROCEDURE 

Subjects  were  greeted,  seated  at  a  comfortable  workstation,  and  were  given  an 
informed  consent  form  (Appendix  E)  to  read  and  sign.  After  consenting  to  participate, 
subjects  were  randomly  assigned  a  unique  identifier  to  track  subjects’  system  generated 
data.  There  was  no  connection  between  this  number  and  identifiable  information  of  the 
subjects. 

Once  assigned  a  number,  participants  took  a  computer-based  survey  to  collect 
demographic  data,  which  was  subsequently  used  to  assign  the  participant  to  one  of  the 
experimental  groups.  This  was  done  in  order  to  ensure  that  the  two  groups  were  balanced 
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in  terms  of  number  of  participants  and  mean  experience  score  (detailed  explanation  on 
the  process  used  can  be  found  in  the  section  IV. E).  Once  assigned  to  a  group,  the 
experimenter  read  this  text  describing  the  scenario  to  the  subject: 

You  are  the  CO  of  a  submarine  located  in  a  patrol  zone,  where  the  transit 
of  an  enemy  task  force  is  expected.  There  is  a  leakage  in  the  main 
circulation  pipe  connector.  Your  engineers  need  low  pressure  in  the  pipes 
so  they  can  fix  the  leakage.  You  also  cannot  use  propulsion  because  the 
cooling  system  for  the  main  shaft  is  secured,  due  to  the  leakage. 
Propulsion  can  be  used  only  in  emergency  case,  for  diving.  You  as  the 
CO,  decide  to  stay  at  PD  as  long  as  you  can,  diving  only  in  case  of 
imminent  danger.  The  submarine  starts  at  PD,  with  all  the  masts  lowered; 
speed  zero  and  heading  north  (Course  000°).  When  the  scenario  starts,  you 
have  already  detected  and  tracked  three  contacts.  They  are  classified  as 
frigates  of  the  Greenhalgh  class  (Type  22).  Contact  1  is  the  frigate  on 
bearing  010°,  Range  6000  yards.  Contact  2  has  bearing  350°,  range  6500 
yards.  Contact  3  is  straight  on  your  bow,  bearing  000°,  range  5500  yards. 

Figure  17  illustrates  the  initial  state  for  the  described  scenario. 


Submarine  and  ships  representations  not  to  scale. 

Figure  17.  Scenario  Initial  Situation. 
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Subjects  then  watched  a  training  video  that  explained  details  of  the  study  and  how 
to  operate  the  STTS.  Since  there  were  differences  in  the  STTS  for  the  different 
conditions,  each  group  received  a  different  video,  which  was  specific  for  their  condition. 
The  complete  transcripts  of  both  videos  are  in  the  Appendix  F. 

After  watching  the  video  tutorials,  participants  were  given  five  minutes  to 
familiarize  themselves  on  a  basic  control  version  of  the  STTS.  After  these  five  minutes  of 
familiarization,  participants  were  redirected  to  the  main  application,  and  asked  to  execute 
three  trials  of  the  GODEX  scenario.  Participants  in  the  PHF  group  had  three  minutes  after 
the  first  and  second  runs  to  read  the  feedback  generated  for  each  run,  which  was 
presented  to  the  participant  on  the  screen  as  text  file.  Participants  in  RTF  group  received 
feedback  during  their  first  two  trials.  Neither  group  received  feedback  during  the  third 
trial  so  that  the  third  runs  were  comparable  between  groups. 

After  completing  the  main  exercise,  the  participants  completed  a  post  task  survey 
(see  Appendix  G). 
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VI.  EXPERIMENT  RESULTS 


A.  SUBJECTS  SCREENING 

A  total  of  40  participants  volunteered  and  were  enrolled  in  the  experiment.  In  the 
early  stages  of  the  experiment,  a  software  bug  causing  the  participants  on  the  PHF  group 
to  have  improper  scores  was  detected.  The  bug  was  promptly  fixed;  however,  data 
generated  for  five  participants  could  not  be  included  in  the  final  analysis.  One  participant 
used  improper  tactics  to  achieve  an  artificially  high  score  and  these  data  were  excluded 
from  the  analysis.  The  next  chapter  discusses  this  case  in  detail.  In  total,  34  valid  cases 
were  analyzed  for  this  investigation.  The  median  age  for  the  participants  was  32,  while 
the  median  time  of  military  service  for  those  in  the  military  was  10  years.  Demographics 
for  the  all  subjects  are  shown  in  Figure  18. 

Demographic  information  was  used  to  place  each  subject  into  one  or  the  other 
treatment  group  (RTF  or  PHF)  in  a  manner  described  in  section  IV. E  in  order  to  maintain 
an  equivalent  level  of  experience  in  each  group.  The  demographics  for  each  group  are 
shown  in  Figure  19  and  Figure  20. 
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Figure  18.  Participant  Demographics. 


Figure  19.  Demographic  Factors:  PHF  Group. 
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Figure  20.  Demographic  Factors:  RTF  Group. 


B.  STATISTICAL  ANALYSIS 

IMP  (Version  12.0.1;  SAS  Institute,  2015)  was  used  to  conduct  all  statistical 
analysis  for  this  study. 

For  the  statistical  analysis  of  the  results,  three  different  response  variables  were 

used: 


•  Observation  Time:  the  mean  time  that  the  participant  spent  with  the 
periscope  raised  for  all  observations  in  a  trial. 

•  Raw  Score:  the  total  of  points  accumulated  during  one  trial.  The  raw  score 
calculation  consists  of  all  correct  and  incorrect  actions  rights  and  wrongs 
committed  by  the  participant  during  the  trial. 

•  Adjusted  Score:  this  response  variable  represents  overall  performance 
combines  both  speed  and  precision  factors.  Figure  21  illustrates  a  surface 
plot  with  participants’  mean  adjusted  scores,  mean  raw  scores  and  mean 
observation  times. 
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Figure  21.  Surface  Plot  of  the  Three  Response  Variables. 

The  variables  analyzed  as  influence  factor  for  the  response  variables  are 

•  Group  (Categorical  -  PHF  or  RTF) 

•  Gamer  (Categorical  -  YES  or  NO); 

•  Military  (Categorical  -  YES  or  NO); 

•  Navy  (Categorical  -  YES  or  NO); 

•  Submariner  (Categorical  -  YES  or  NO); 

•  Years  of  Service  (YOS)  (Categorical  -  YES  or  NO) 

•  Gender  (Categorical  -  MAEE  or  EEMAEE) 

•  Preferred  hand  for  writing  (Categorical  -  RIGHT  or  EEET) 

•  Trial  (Interval  -  1,  2,  or  3) 
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1. 


Research  Question  1 


Research  question  1  was  “Can  a  game-based,  part-task  periscope  simulation  that 
incorporates  the  appropriate  cues  to  train  identified  knowledge,  skills,  and  abilities  be 
developed  to  supplement  periscope  operator  training?”  Paired-samples  Mests  were  used 
to  verify  the  improvements  on  adjusted  scores  achieved  by  the  34  participants  throughout 
the  three  trials  of  the  experiment.  Results  indicated  that  performance  improved 
significantly  for  each  trial.  See  Table  5  and  Figure  22.  Detailed  results  can  be  found  in 
Appendix  H. 


Table  5.  Paired-Samples  t-Test  Results  for  Participants’  Adjusted  Scores. 


M 

SD 

t 

Sig.  (2-tailed) 

Trial  1 

3146.36 

1553.38 

6.99 

<  .0001 

Trial  2 

4826.28 

2215.14 

Trial  1 

3146.36 

1553.38 

7.48 

<  .0001 

Trial  3 

6024.04 

2943.65 

Trial  2 

4826.28 

2215.14 

4.00 

.0003 

Trial  3 

6024.04 

2943.65 

8000  -I - 

■  Trial  1 

■  Trial  2 

■  Trial  3 


Figure  22.  Adjusted  Scores  by  Trial. 

2.  Research  Question  2 

Research  question  2  was  “Do  differences  in  trainee  performance  exist  when 

exposed  to  different  feedback  types  over  the  course  of  multiple  sessions?”  Paired- 

samples  Mests  were  used  to  analyze  adjusted  scores  improvements  for  each  condition 

(RTF  and  PHF).  Results  indicated  that  performance  improved  significantly  for  each  trial 

for  RTF  and  PHF  (see  Figure  23).  Table  6  (PHF)  and  Table  7  (RTF)  show  the  results  for 

adjusted  score  for  each  group.  Detailed  results  are  shown  in  Appendix  I. 
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Table  6.  Paired-Samples  r-Test  Results  for  Participants’  Adjusted  Scores  in 

the  PHF  Group. 


M 

SD 

t 

Sig.  (2-tailed) 

Trial  1 

2798.60 

1441.74 

6.89 

<  .0001 

Trial  2 

4899.35 

2062.07 

Trial  1 

2798.60 

1441.74 

7.14 

<  .0001 

Trial  3 

6178.79 

2496.86 

Trial  2 

4899.35 

2062.07 

3.48 

.003 

Trial  3 

6178.79 

2496.86 

Table  7.  Paired-Samples  t-Test  Results  for  Participants’  Adjusted  Scores  in 

the  RTF  Group. 


M 

SD 

t 

Sig.  (2-tailed) 

Trial  1 

3494.11 

1625.17 

3.59 

.002 

Trial  2 

4753.20 

2420.10 

Trial  1 

3494.11 

1625.17 

3.98 

.001 

Trial  3 

5869.29 

3403.90 

Trial  2 

4753.20 

2420.10 

2.31 

.034 

Trial  3 

5869.29 

3403.90 

■  ■Trial  1 

■  Trial  2 
■  Trials 

- 1 

RTF 

Figure  23.  Adjusted  Scores  by  Trial,  by  Group. 

3.  Exploratory  Investigation  of  Research  Question  2 

Based  upon  the  observation  of  the  results  presented  above,  an  exploratory  analysis 
was  conducted  to  determine  if  differences  existed  between  groups  across  trials.  When 
comparing  performance  achieved  by  participants  in  each  group  for  each  trial,  a 
significant  difference  in  adjusted  scores  between  PHF  group  (M  =  2586.17,  SZ)=1182.75) 
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and  RTF  group  (M  =  3494.11,  SD  =  1625.17),  t(29.20)  =  1.84,p  =  0.0378  were  observed 
for  trial  one  (see  Figure  24).  Significant  differences  between  PHF  and  RTF  were  not 
observed  for  trials  two  and  three. 


Trial  1  Trial  2  Trial  3 

Figure  24.  Adjusted  Scores  Compared  between  Groups  per  Trial. 

In  order  to  visually  observe  raw  score  (related  to  the  precision  of  observations) 
and  observation  time  (related  to  participants’  speed),  these  variables  were  plotted  on  the 
graph  shown  in  Figure  25.  Mean  raw  score  (Y-axis)  is  compared  to  observation  time  (X- 
axis)  and  four  quadrants  were  constructed  around  the  medians — 7125  points  and  25.64 
seconds,  respectively.  Quadrant  I  shows  the  lowest  possible  adjusted  scores  (high 
observation  times  and  low  raw  scores).  Data  points  in  quadrant  II  represent  short 
observation  times  and  low  raw  scores,  while  quadrant  III  shows  long  observation  times 
and  high  raw  scores.  The  best  performance  is  shown  in  quadrant  IV  and  represents 
participants  with  short  observation  times  and  high  raw  scores. 
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Figure  25.  Subjects  Distributed  on  Quadrants  Based  on  Mean  Results  for  the 

Three  Trials. 


By  observing  participants’  evolutions  through  the  quadrants  trial  by  trial,  it  is 
possible  to  observe  that  there  was  a  general  movement  from  Q-I  to  Q-IV,  confirming  the 
conclusion  that  performance  incrementally  improved  through  each  trial  (see  Table  8). 


Table  8.  Quadrant  Distribution  across  Trials. 


QUADRANT 

TRIAL  I 

TRIAL  2 

TRIAL  3 

QI 

17 

II 

7 

QII 

3 

5 

7 

QIII 

10 

7 

3 

QIV 

4 

II 

17 
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4.  Research  Question  3 

Research  question  3  was  “What,  if  any,  differences  exist  between  real-time 
feedback  (RTF)  and  post-hoc  feedback  (PHF)  on  final  trainee  performance?”  The 
experiment  was  designed  to  allow  the  comparison  between  treatment  groups.  Both  PHF 
and  RTF  subjects  received  their  specific  feedback  for  the  first  and  second  trials.  The  third 
trial  for  both  groups  was  the  same,  with  no  feedback  provided,  and  thus  provides  a  more 
equivalent  representation  of  the  performance  of  each  group.  Therefore,  the  third  trial  was 
used  for  all  further  analysis  involving  comparisons  between  groups. 

A  t-test  was  used  to  compare  the  adjusted  score  achieved  for  each  group  and 
revealed  that  there  was  not  enough  evidence  to  reject  the  null  hypothesis  H3o.  There  was 
no  significant  difference  between  the  adjusted  scores  achieved  from  participants  in  PHF 
group  (M  =  6178.79,  5D=2496.89)  and  RTF  group  (M  =  5896.29,  SD  =  3403.90), 
t(29.35)  =  -0.3023,  p  =  0.7646  (see  Figure  26). 
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OnewayAnalysisof  ADJ  SCORE  RUN3  By  GROUP 

14000 
z  12000 

i  10000 
s  8000 
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y 

1  y 

PHF  RTF 


GROUP 


Quantiles 

Level 

Minimum 

10% 

25% 

Median 

75% 

90%  Maximum 

PHF 

3212.387 

351648 

43  53.645 

5554.447 

7720.583 

10039.78 

1309344 

RTF 

1744.809 

1998.616 

3001262 

4703.991 

9072266 

11248.7 

1251028 

Means  and  Std  Deviations 

Std  Err 

Level 

Number 

Mean 

Std  Dev 

Mean  Lower  95% 

Upper  95% 

PHF 

17 

6178.79 

249686 

605.58 

4895.0 

7462.6 

RTF 

17 

586929 

3403.90 

825.57 

41192 

76194 

tTest 

RTF-PHF 

Assuming  unequal  variances 
Difference  -309.5  t  Ratio  -0.30228 

StdErrDif  1023.9  DF  29.35238 

Upper  CL  Dif  17834  Prob  >  |t|  0.7646 

Lower  CL  Dif  -24024  Prob  >  t  0.6177 

Confidence  0.95  Prob  <  t  0.3823  -4000  -2000  0  1  000  3000 


Figure  26.  Adjusted  Seores  on  Trial  3  Compared  between  Groups. 


5.  Exploratory  Investigation  on  Research  Question  3 

As  no  significant  performance  differences  were  observed  for  trial  three,  additional 
analysis  were  conducted  to  potentially  observe  effects  of  other  contributing  factors. 

To  determine  the  effects  of  demographic  factors  on  adjusted  score  achieved  in 
trial  3,  a  linear  regression  fit  was  performed  across  all  34  participants.  The  factors 
military,  submariner,  gender,  and  handedness  were  not  considered  in  the  analysis  across 
groups  due  to  their  small  representation  in  this  dataset.  None  of  the  factors  was 
statistically  significant  to  the  model  for  the  PHF  group.  On  the  RTF  group,  the  factors 
age  ip  =  0.005),  gamer  (p  =  0.02),  and  the  combination  navy  *  YOS  (p  =  0.013)  were 
included  as  significant  for  the  regression  model,  F(6,10)  =  4.42,  p  =  0.019,  =  .073  with 

an  adjusted  R  =.561.  Figure  27  details  the  elements  of  this  analysis. 
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Response  ADJ  SCORE  RUN3  GROUP=RTF 
Actual  by  Predicted  Plot 


ADJ  SCORE  RUN3  Predicted  P=0.0194  RSq=0.73 
RMSE=2253.7 


Summary  of  Fit 

RSquare  0,72602 

RSquareAdj  0.561631 

Root  Mean  Square  Error  2253.703 

Mean  of  Response  5869.289 

Observations  (or  Sum  Wgts)  17 

Analysis  of  Variance 

Sum  of 

Source  DF  Squares  Mean  Square  F  Ratio 

Model  6  1  34592934  22432156  4.4165 

Error  10  50791786  5079178,6  Prob  >  F 

C  Total  16  1  85384720  0.0194- 


Sorted  Parameter  Estimates 


Term 

Estimate 

Std  Error 

t  Ratk) 

Prob>|t| 

AGE(0] 

Biased 

4934.9046 

1380.106 

3.58 

0.0050* 

GAMER(0] 

Biased 

-2613.384 

948.387 

-2,76 

r 

0.0203* 

NAVY(0]*YOS[0) 

Biased 

1783.6586 

690.0529 

2.58 

p 

0.0272* 

YOS[0] 

Biased 

-2858A34 

1321,351 

-2.16 

0,0558 

NAVY[0] 

Biased 

2297.0891 

1078.878 

2.13 

p . 

0,0591 

GAMER[0]*NAVY[0] 

Biased 

-917.6643 

948,387 

-0.97 

0,3561 

GAMERtOl-YOSIO] 

Zeroed 

0 

0 

GAMER[0]’AGE[01 

Zeroed 

0 

0 

NAWI0]*AGE[0] 

Zeroed 

0 

0 

YOS[0]*AG£(0] 

Zeroed 

0 

0 

Figure  27.  Effects  of  Demographic  Factors  on  Adjusted  Scores. 


An  additional  two-tailed  t-tests  comparing  both  response  variables  (raw  scores 
and  observation  times)  by  the  factors  (gaming  experience  and  age)  was  performed  to 
determine  if  age  and  gamer  experience  contributed  to  the  results  above.  Results  showed 
statistically  significant  difference  of  raw  scores  between  gamers  {M  =  9066.61,  SD  = 
292.97)  and  non-gamers,  t(14.94)  =  2.41,  p  =  0.029  (see  Figure  27).  Statistically 
significant  difference  were  also  found  for  observation  times  for  non-gamers  (M  =  26.01, 
SD  =  4.89)  and  gamers  (M  =  21.53,  SD  =  2.17),  til .31)  =  -2.47,  p  =  0.041;  and  for  age 
group  32  years  or  less  (M  =  22.54,  SD  =  3.28)  and  age  group  above  32  years  (M  =  27.10, 
SD  =  4.95),  t(14.98)  =  2.28,  p  =  0.037  (see  Figures  28  and  29). 
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Oneway  Analysis  of  RAW  SCORE 
RUN3  By  GAMER  GROUP=RTF 


GAMER 


Means  and  Std  Deviations 

Level  Number 

Mean 

Std  Dev 

Std  Err 

Mean 

Lower  95% 

Upper  95% 

0  14 

1  3 

7853.57 

9066.67 

177444 

292.97 

47424 

169.15 

6829.0 

8338.9 

8878.1 

9794.5 

tTest 

1-0 


Assuming  unequal  variances 
Difference  1213.10 

StdErrDif  503.50 

Upper  CL  Dif  2286.62 

Lower  CL  Dif  139.57 

Confidence  0.95 


t  Ratio 
DF 

Prob  >  |t| 
Prob  >  t 
Prob  <  t 


2.409317 

14.94578 

0.0293* 

0.0147* 

0.9853 


Figure  28.  Statistically  Significant  Difference  on  Raw  Scores  for  Gamers. 


Fit  Group  GROUP=RTF 

Oneway  Analysis  of  OBS  TIME  RUN3  By  GAMER  GROUP=  RTF 


Means  and  Std  Deviations 

Std  Err 

Level  Number  Mean  Std  Dev  Mean  Lower  95%  Upper  95% 
0  1  4  26.0107  4.89285  1.3077  23,186  28.836 

1  3  21.5293  2.17355  1.2549  16,130  26,929 

tTest 

1-0 


Assuming  unequal  variances 


Difference 

-4,4814  t  Ratio 

-2.47261 

Std  Err  Dif 

1.8124  DF 

7,365561 

Upper  CL  Dif 

-0,2385  Prob  >  |t| 

0.0410’ 

Lower  CL  Dif 

-8.7242  Prob  >  t 

0.9795 

Confidence 

0.95  Prob  <t 

0.020? 

OnewayAnalysisofOBSTIMERUNS  By  AGE  GROUP=RTF 


Means  and  Std  Deviations 

Std  Err 

Level  Number  Mean  Std  Dev  Mean  Lower  95%  Upper  95% 

0  7  22,5389  3.28295  1.2408  1  9.503  25,575 

1  10  27.0965  4.94513  1.5638  23,559  30.634 

tTest 

1-0 


Assuming  unequal  variances 


Difference 

4.55755  t  Ratio 

2.283029 

Std  Err  Dif 

1.99627  DF 

14.98838 

Upper  CL  Dif 

8.81280  Prob  >  |t| 

0,0374* 

Lower  CL  Dif 

0.30231  Prob  >  t 

0.0187* 

Confidence 

0.95  Prob  <  t 

0.9813 

Figure  29.  Statistically  Significant  Difference  on  Observation  Times  for  Both 

Gamers  and  Participants  <=32  Years  Old. 
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VII.  DISCUSSION 


The  overarching  goal  of  this  thesis  was  to  develop  a  prototypical  training  tool  for 
submarine  officers  and  to  determine  if  use  of  the  tool  could  improve  performance. 
Following  an  ISD  process,  RQl  was  answered  by  the  analysis,  design,  development,  and 
implementation  of  the  STTS.  Further,  the  results  of  paired  t-tests  comparing  performance 
scores  across  the  three  trials  demonstrate  that  users  of  the  system  did  experience  a 
learning  effect  on  the  simulated  GODEX  task.  These  results  should  be  considered  in 
context  of  the  conditions  of  the  experiment  and  that  performance  gains  shown  here  may 
not  guarantee  improved  performance  on  an  actual  GODEX  mission. 

The  performance  increase  for  each  group  during  the  experiment  (RQ2)  was 
assessed  by  paired  t- tests  comparing  adjusted  scores  for  each  trial.  The  significant 
increase  of  adjusted  scores  for  both  groups  taken  with  the  exploratory  analysis  which 
showed  significant  difference  of  performance  between  groups  only  on  trial  one  (RTE  > 
PHE),  can  be  attributed  to  the  total  lack  of  feedback  received  by  the  PHE  group  during 
the  first  trial.  Participants  receiving  RTE  higher  performance  scores  in  trial  one  are  likely 
due  to  the  feedback  they  received  during  that  trial.  These  results  support  the  notion  that 
RTE  can  be  used,  at  least  initially,  to  improve  performance  under  conditions  similar  to 
those  used  in  this  study. 

To  determine  whether  different  types  of  feedback  would  affect  the  performance  of 
participants  (RQ3),  the  mean  scores  obtained  by  participants  in  each  group  were 
compared.  There  was  no  statistically  significant  difference  for  the  third  trial  (trial  used  to 
make  comparisons  across  groups). 

The  simple  comparison  of  mean  adjusted  scores  for  RQ3  was  taken  to  be  non- 
conclusive.  An  exploratory  analysis  provided  a  deeper  understanding  about  the  effect  of 
demographic  factors  on  adjusted  scores.  Participants  who  received  RTE  and  had  no¬ 
gaming  background  had  a  significant  decrement  in  their  performance  compared  to 
gamers.  Eurther  analysis  showed  that  there  was  a  significant  increase  on  observation  time 
(negatively  affecting  the  adjusted  score).  Additionally,  gamers  exposed  to  RTE,  were 
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more  precise  in  their  observations  thus  achieving  better  raw  scores.  The  age  factor 
affected  the  observation  time,  indicating  that  participants  older  than  32  who  were 
provided  with  RTF  had  longer  observation  times,  indicating  that  this  type  of  feedback  can 
be  distracting.  These  results  suggest  that  age,  experience,  and  behavioral  factors  should 
be  considered  when  deciding  the  feedback  type  that  training  participants  should  receive 
when  developing  a  system  for  training.  Further,  using  the  background  and  age  of 
participants,  an  intelligent  tutoring  system  should  be  able  to  deliver  only  the  necessary 
amount  of  information,  in  the  correct  moment  during  training  events. 

As  noted  in  section  VI.A.,  one  subject  employed  an  approach  to  maximize  the 
STTS  score  that  was  not  consistent  with  the  instructions  of  the  game.  After  trial  1,  the 
subject  decided  to  ignore  the  lateral  targets  and  focus  solely  on  the  center  contact.  The 
decision  to  not  observe  the  other  contacts  allowed  the  observation  time  for  each  become 
negative.  These  actions  resulted  in  the  subject  being  able  to  make  fewer  observations. 
This  resulted  in  a  better  SIR  and  a  much  higher  adjusted  score.  The  raw  score  was  not 
highly  impacted  because  the  scoring  rules  were  defined  so  these  actions  would  result  in 
only  the  loss  of  points  one  time  for  each  contact  that  has  its  look  interval  exceeded.  As 
stated  above,  data  from  this  trial  were  not  included  in  the  final  analysis  due  to  the  fact 
that  the  participant  deliberately  disobeyed  the  rules  as  they  applied  to  submariners  in 
order  to  maximize  the  adjusted  score. 

Game  developers,  especially  serious  game  developers,  must  thoughtfully  create 
rules  and  scoring  approaches.  If  subverting  the  game’s  intent  to  better  achieve  the  goal 
does  not  incur  a  penalty  from  the  rules,  users  are  likely  to  discover  and  exploit  potential 
loopholes.  Game  designers  need  to  be  very  careful  that  their  games  do  not  contain  such 
“cheats,”  since  it  can  lead  to  negative  training,  and  they  must  constantly  tune  play  to 
ensure  only  correct  real-world  behaviors  result  in  acceptable  scores  in  the  game  or 
simulator. 
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VIII.  CONCLUSION  AND  FUTURE  WORK 


A.  CONCLUSION 

The  prototype  developed  for  this  thesis  is  an  initial  prototype  that  can  be  the  basis 
for  future  training  systems  for  the  BNSS.  Additionally,  the  results  presented  here 
demonstrate  the  value  of  automated  feedback  to  performance.  Of  equal  importance,  when 
designing  adaptive  or  intelligent  training  systems,  understanding  demographics  of  the 
training  audience  should  be  considered  when  determining  the  type  of  feedback  to 
employ.  Knowing  the  differences  in  efficacy  of  feedback  type  for  different  populations 
will  aid  in  the  appropriate  selection  feedback  for  the  desired  training. 

Clearly,  affordable  part-task  training  solutions  that  incorporate  automated 
feedback  such  as  the  STTS  can  be  rapidly  developed  by  following  a  sound  ISD  process. 
Future  instructional  systems  designers  can  benefit  from  the  results  and  conclusions  of  this 
thesis.  Further,  the  development  of  the  STTS  was  supported  by  the  extensive  features 
available  in  the  Unity  community,  the  use  of  external  assets  and  the  range  of  support, 
which  can  be  found  online,  for  beginners  and  experts  alike,  resulted  in  a  steep  learning 
curve.  This  resulted  in  a  complete  simulation,  fully  functional  for  the  purpose  of  this 
thesis,  in  only  few  months.  This  accomplishment  demonstrates  that  the  rapid  innovation 
of  systems  is  now  firmly  in  the  realm  of  the  possible.  Ultimately,  this  may  mean  that  the 
approach  to  designing,  validating  and  acquiring  new  training  systems  and  programs  may 
need  to  be  modified  in  order  to  take  advantage  of  these  advances. 

B.  FUTURE  WORK 

In  order  to  be  validated  as  a  training  tool,  the  STTS  and  concepts  explored  here 
will  require  modifications  and  additional  transfer  of  training  research.  The  ability  to  tailor 
feedback,  by  the  system,  instructor  and/or  trainee  could  provide  improved  training  results 
and  these  areas  should  be  isolated  and  explored  in  a  controlled  environment.  It  is 
important  to  note  that  for  this  study,  RTF  was  provided  as  pop-up  text  messages  that  were 
viewable  for  three  seconds  on  the  main  screen  of  the  simulation  for  each  action  taken. 
Future  research  using  the  STTS  or  other  game -based  simulations  should  consider 
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investigating  the  impact  of  the  number  of  messages  on  performance  during  time 
constrained  tasks.  Additionally,  a  worthy  exercise  would  be  an  investigation  of  how 
pausing  the  simulation  and  providing  longer  periods  of  instructions.  Finally,  the  influence 
of  feedback  modality  on  performance  and  retention  should  be  investigated. 

The  system  developed  for  this  thesis  is  a  prototype  and  a  starting  point  for 
developing  a  real  tactical  training  tool.  The  considerations  made  to  keep  the  STTS  an 
UNCLASS  system  limited  the  scope  of  what  can  be  trained  by  a  real  submarine  officer 
by  this  system.  Future  work  to  adjust  this  base  architecture  to  a  complete  tactical  trainer 
include: 

•  Incorporation  of  realistic  buoyancy  effects  with  a  different  sea  model  and 
realistic  physics,  to  allow  for  the  adjustment  of  varied  conditions  of  sea-states 
and  weather. 

•  The  implementation  of  voice  recognition  in  order  to  make  the  simulator  more 
closely  model  the  SO’s  actions  as  well  as  optimize  the  usability  of  the  GUI 
solution  developed  for  this  thesis. 

•  The  incorporation  of  excluded  TTPs  and  other  types  of  observations  for  the 
GODEX  mission  which  would  need  to  be  included  if  implementing  a 
classified  version  for  use  in  controlled  environments. 

•  The  implementation  of  an  intelligent  tutoring  system,  capable  of  adapting  to 
the  type  and  amount  of  feedback,  based  on  trainee’s  performance,  previous 
experience,  and  age. 

•  A  usability  study  using  a  sample  from  the  population  from  the  intended  users 
of  the  simulation. 

•  Additionally,  this  simulation  could  be  adjusted  for  use  with  tablets, 
smartphones,  and  other  display  technologies  in  order  to  increase  portability 
and  take  advantage  of  built  in  gyroscopes  and  accelerometers  for  control  the 
periscope. 
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APPENDIX  A.  GROUP  ASSIGNMENT  ALGORITHM 


1.  Assign  0  or  1  to  each  of  the  5  variables  on  the  equation’s  right  side  based  on  the 
answers  given  on  the  Demographics  Survey  (0  if  “NO”,  1  if  “YES”); 

experienceScore  =  (gamer  x  15)  +  (military  x  5)  +  (navy  x  10)  +  (sub  x  20)  +  (yos  x  5) 


2.  Read  from  files  the  experiences  of  subjects  in  each  group; 

3.  If  difference  of  number  of  subjects  is  more  than  three,  then  assign  the  subject  to  the 
group  with  fewer  subjects  and  go  to  step  7; 

4.  Average  the  experience  in  each  group  (m  and  n  are  the  number  of  subjects  in  the 
control  and  treatment  groups,  respectively); 

m 

'yexperience(i) 

oldMeanControl  =  — - 

m 


'y^experience(i) 


oldMeanTreat  - 


5.  Find  the  new  means  for  each  group  considering  that  the  subject  was  assigned  to  that 
group; 


'^experience(i)  +  experienceScore 

newMeanControl  =  - 

m  + 1 

n 

'^experience(i)  +  experienceScore 


newMeanTreat 


—  1=1 


n  +  \ 


6.  Assign  the  subject  to  the  group  that  will  generate  a  smaller  difference  between  means; 
If 

(I  oldMeanControl  —  newMeanTreat  kl  oldMeanTreat  —  newMeanControl  I) 

assign  the  subject  to  the  treatment  group; 

(I  oldMeanControl  -  newMeanTreat  l>l  oldMeanTreat  -  newMeanControl  I) 

assign  the  subject  to  the  control  group; 

(I  oldMeanControl  -  newMeanTreat  1=1  oldMeanTreat  -  newMeanControl  I) 

assign  the  subject  to  the  group  with  fewer  subjects; 

7.  Write  a  new  line  in  the  correspondent  file  with  subject  number  and  his/her 
experienceScore; 

8.  Setup  the  simulation  to  present  feedback  accordingly  to  the  group  chosen. 
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APPENDIX  B.  NPS  IRB  PROTOCOL  APPROVAL 


Naval  Postgraduate  School 
Human  Research  Protection  Program 


From:  President,  Naval  Postgraduate  School  (NPS) 

To:  LT  Lee  Sciarini,  USN 

Mr.  Perry  McDowell,  Modeling,  Virtual  Environments  and 
Simulation  Institute  (MOVES) 

LT  Rodrigo  da  Silva  Vieira,  USN 
Via:  Vice  Chairman,  Institutional  Review  Board  (IRB) 

SUBJ:  GAME-BASED  PERISCOPE  SIMULATOR  FOR  SUBMARINE  OFFICERS 

TACTICAL  TRAINING 

Enel:  (1)  Approved  IRB  Initial  Review  Protocol 

1 .  The  NPS  IRB  is  pleased  to  inform  you  that  the  NPS  President 
has  approved  your  initial  review  protocol  (NPS  IRB# 

NPS .2016 . OOS1-IR-EP7-A) .  The  approved  IRB  Protocol  is  found  in 
enclosure  (1) .  Completion  of  the  CITI  Research  Ethics  Training 
has  been  confirmed. 

2.  This  approval  expires  on  30  June  2016.  If  additional  time 
is  required  to  complete  the  research,  a  continuing  review  report 
must  be  approved  by  the  IRB  and  NPS  President  prior  to  the 
expiration  of  approval.  At  expiration  all  research  (subject 
recruitment,  data  collection,  analysis  of  data  containing  PII) 
must  cease. 

3.  You  are  required  to  obtain  documented  consent  as  outlined  in 
the  approval  protocol . 

4.  You  are  required  to  report  to  the  IRB  any  unanticipated 
problems  or  serious  adverse  events  to  the  NPS  IRB  within  24 
hours  of  the  occurrence. 

5.  Any  proposed  changes  in  IRB  approved  research  must  be 
reviewed  and  approved  by  the  NPS  IRB  and  NPS  President  prior  to 
implementation  except  where  necessary  to  eliminate  apparent 
immediate  hazards  to  research  participants  and  subjects. 

6.  As  the  Principal  Investigator  (PI)  it  is  your  responsibility 
to  ensure  that  the  research  and  the  actions  of  all  project 
personnel  involved  in  conducting  this  study  will  conform  with 
the  IRB  approved  protocol  and  IRB  requirements/policies. 
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Sub  j  : 


GAME-BASED  PERISCOPE  SIMULATOR  FOR  SUBMARINE  OFFICERS 
TACTICAL  TRAINING 


7.  At  completion  of  the  research,  no  later  than  expiration  of 
approval,  the  PI  will  close  the  protocol  by  submitting  an  End  of 
Experiment  Report . 


K.  J.  Euske 


Oigitairy  signed  by  K.  J.  Euske 
DN;  cn=K.  J.  Euske.  o«NPS. 
ou>GSBPP. 

ema(]=k^skeCinps  edu,  c^US 
Date:  2016  04.04  15;06  30  -07*00’ 


Kenneth  J.  Euske,  PhD 
Vice  Chair 

Institutional  Review  Board 


Ronald  A.  Route 

Vice  Admiral,  U.S.  Navy  (Ret.) 

President,  Naval  Postgraduate  School 

Date :  ^ 
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APPENDIX  C.  RECRUITMENT  POSTER 


Submarine  T ac±ical  T raining  System 


Input  RarMM 


SCORE  adjusted: 


Help  a  student's  thesis  research  to  understand  the  impact  of  two  different  types  of 
feedbacks  in  a  trainee’s  performance. 

You  will  be  trained  in  a  3D  Game-Based  simulation  of  a  submarine  periscope,  while 
acting  as  the  Submarine’s  Commanding  Officer.  Your  goal  is  to  keep  the  submarine 
safely  in  the  periscope  depth  while  surface  contacts  approach  in  collision  route. 


WHO:  NPS  students,  faculty  or  staff  members 
(Active  Duty,  retired  or  civilians) 

WHERE:  Watkins  212-A  (NPS  Vision  Lab) 
WHEN:  Choose  any  time  you  can  spare  1-1.5  hour 
CONTACT:  LCDR  Rodrigo  Vieira  »t  rdasilva&nps.edu 


Risks  associated  with  this  study  are  minimal.  Participation  is  voluntary.  The 
point  of  contact  for  risk  questions  or  concerns  is  Dr.  Lee  Sciarini 
Uv/sciarilS)nDs. edu).  Prospective  subjects  may  contact  NPS  iRB  Vice  -  Chair,  Dr. 
Ken  Euske,  831-656-2860.  kieuske&nos.edu  regarding  rights  as  a  participant. 
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APPENDIX  D.  RECRUITMENT  SCRIPT 


"Game-based  Periscope  Simulator  for  Submarine  Officers  Tactical  Training" 

Good  Morning, 

Curious  about  We  are  conducting  an  experiment  as  part  of  a  research  study  to  increase 
our  understanding  in  how  feedback  can  impact  the  overall  performance  during  training. 

As  a  participant,  you  will  use  the  "Submarine  Tactical  Training  Simulation"  which  was 
developed  to  support  the  experiment. 

We  recognize  how  busy  you  are.  Your  participation  will  support  the  effort  of  a  thesis 
student,  and  we  appreciate  your  willingness  to  assist  in  this  study. 

If  you  decide  to  participate,  you  will  act  as  a  Submarine's  Commanding  Officer  and  your 
goal  will  be  to  keep  the  submarine  safely  in  the  periscope  depth  while  surface  contacts 
approach  in  collision  route. 

All  participants  will  be  provided  detailed  instructions  on  the  use  of  the  simulation  and  the 
simulated  task  you  will  be  asked  to  perform. 

The  experiment  will  take  from  1  to  1.5  hours. 

Risks  associated  with  this  study  are  minimal.  Participation  is  voluntary.  The  point  of 
contact  for  risk  questions  or  concerns  is  the  Principal  Investigator,  Dr.  Lee  Sciarini 
(lwsciari@nps.edu).  Prospective  subjects  may  contact  NPS  IRB  Vice  -  Chair,  Dr.  Ken  Euske 
(831-656-2860  -  i<ieusi<e@nps.edu)  regarding  rights  as  a  participant. 

All  the  data  collected  will  be  kept  confidential.  Each  participant  will  be  assigned  a  number 
code  to  help  ensure  that  personal  identifiers  are  not  revealed  during  the  analysis  and  write 
up  of  findings. 

There  is  no  compensation  for  participating  in  this  study.  However,  your  participation  will 
be  a  valuable  addition  to  our  research  and  findings  could  lead  to  greater  understanding  of 
feedback  as  tools  for  training  and  education  systems. 

If  you  are  willing  to  participate  please  suggest  a  day  and  time  that  suits  you  and  I'll  do  my 
best  to  be  available.  If  you  have  any  questions  please  do  not  hesitate  to  ask. 

The  experiment  poster  is  attached. 

Thanks! 

Rodrigo  da  Silva  Vieira 
Lieutenant  Commander 
Brazilian  Navy  Submarine  Officer 
MOVES  Department 
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APPENDIX  E.  INEORMED  CONSENT  EORM 


Naval  Postgraduate  School 
Consent  to  Participate  in  Research 

Introduction.  You  are  invited  to  participate  in  a  research  study  entitled  “Game -based 
Periscope  Simulator  for  Submarine  Officers  Tactical  Training”.  The  purpose  of  the 
research  is  to  analyze  the  impact  of  different  types  of  feedbacks  over  the  performance  of 
a  subject  in  a  military  training  context. 

Procedures.  30  -  40  NPS  students,  faculty  or  staff  (military  or  not)  will  complete  an 
operative  task  as  a  submarine  Captain.  The  objective  of  the  task  is  the  maintaining  the 
safety  of  the  submarine  and  crew  while  staying  in  the  periscope  depth  as  surface 
warships  approach.  Stopwatches  will  display  intercept  times  for  approaching  ships  (time 
remaining  for  the  next  observation  of  that  contact).  You  will  prioritize  observations  by 
remaining  time  to  intercept  and  execute  the  “Go  deep”  command  (dive)  when  the  closest 
contact  reaches  a  critical  point.  You  will  be  asked  to  complete  this  task  three  times,  with 
8-10  minutes  expected  for  each  trial.  You  also  will  be  asked  to  complete  a  demographic 
survey,  and  a  post-task  survey.  The  duration  of  your  participation  will  be  approximately 
1-1.5  hours.  The  simulation  used  and  procedures  described  are  new  and  are  only  related 
to  the  research  and  serve  no  purpose  other  than  this  research  endeavor. 

Location.  The  study  will  take  place  at  NPS,  Watkins  212a. 

Cost.  There  is  no  cost  to  participate  in  this  research  study. 

Voluntary  Nature  of  the  Study.  Your  participation  in  this  study  is  strictly  voluntary.  If 
you  choose  to  participate  you  can  change  your  mind  at  any  time  and  withdraw  from  the 
study.  You  will  not  be  penalized  in  any  way  or  lose  any  benefits  to  which  you  would 
otherwise  be  entitled  if  you  choose  not  to  participate  in  this  study  or  to  withdraw.  The 
alternative  to  participating  in  the  research  is  to  not  participate  in  the  research. 

Potential  Risks  and  Discomforts.  We  anticipate  no  to  very  minimal  discomfort. 
Participants  will  sit  in  an  air-conditioned  room  on  a  comfortable  chair  for  about  1-1.5 
hours  while  they  complete  the  computerized  tasks  and  answer  the  surveys.  Breach  of 
confidentiality  is  a  risk. 

Anticipated  Benefits.  If  this  study  proves  that  one  type  of  feedback  is  significantly 
better  than  the  other,  future  training  and  education  simulations  can  benefit  from  this 
knowledge  and  the  methods  of  training  intervention  can  be  optimized.  You  may  gain 
insight  into  basic  submarine  operations  and  if  you  are  a  submariner  you  can  benefit  from 
this  basic  training  prototype. 

Compensation  for  Participation.  No  tangible  compensation  will  be  given. _ 
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Confidentiality  &  Privacy  Act.  Any  information  that  is  obtained  during  this  study  will 
be  kept  confidential  to  the  full  extent  permitted  by  law.  All  efforts,  within  reason,  will  be 
made  to  keep  your  personal  information  in  your  research  record  confidential  but  total 
confidentiality  cannot  be  guaranteed.  In  accordance  with  NPS  data  storage  instruction, 
data  will  only  be  kept  on  approved  NPS  Systems.  Hard  copies  of  informed  consent  forms 
will  be  kept  in  a  locked  container  behind  a  locked  door.  To  protect  against  disclosure  of 
any  information  all  participants  will  be  assigned  a  random  identification  number  and  will 
only  be  identified  on  surveys  by  that  number.  All  the  scores  gained  by  the  subject  while 
operating  the  simulation  will  only  be  associated  with  that  random  number  as  well.  All 
demographic  data  will  be  kept  secure  in  a  password  protected  computer. 

Points  of  Contact.  If  you  have  any  questions  or  comments  about  the  research,  or  you 
experience  an  injury  or  have  questions  about  any  discomforts  that  you  experience  while 
taking  part  in  this  study  please  contact  the  Principal  Investigator,  Dr.  Lee  Sciarini  at  831- 
656-7675  or  lwsciari@nps.edu.  Questions  about  your  rights  as  a  research  subject  or  any 
other  concerns  may  be  addressed  to  the  Naval  Postgraduate  School  IRB  Vice  -  Chair,  Dr. 
Ken  Euske,  831-656-2860,  kieuske@nps.edu. 

Statement  of  Consent.  I  have  read  the  information  provided  above.  I  have  been  given  the 
opportunity  to  ask  questions  and  all  the  questions  have  been  answered  to  my  satisfaction.  I 
have  been  provided  a  copy  of  this  form  for  my  records  and  I  agree  to  participate  in  this 
study.  I  understand  that  by  agreeing  to  participate  in  this  research  and  signing  this  form,  I 
do  not  waive  any  of  my  legal  rights. 


Participant’s  Signature  Date 


Researcher’s  Signature  Date 
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APPENDIX  F.  VIDEO  TUTORIAL  SCRIPTS 


VIDEO  TUTORIAL  SCRIPT 
REAL-TIME  EEEDBACK  GROUP 


Welcome  to  the  SUBMARINE  TACTICAL  TRAINING  SYSTEM,  and  thank  you 
for  applying  to  this  experiment.  This  is  the  MAIN  SCREEN.  In  the  LEET  PANEL  you 
will  see  INTERIOR  OE  THE  SUBMARINE,  if  the  periscope  is  DOWN  or  the 
OCEANIC  ENVIRONMENT  outside  the  submarine,  if  the  periscope  is  UP.  The 
CENTER  PANEL  is  called  INPUT  PANEL  and  it  will  be  used  to  control  the  periscope, 
give  orders  and  provide  information  about  the  observations.  The  RIGHT  PANEL  is 
called  the  EEEDBACK  PANEL  and  it  will  be  used  to  show  the  SCORES  and  a  log  of  the 
feedback  received  before. 

One  observation  consists  of  this  sequence: 

1.  CHOOSE  the  contact  to  observe; 

2.  RAISE  the  periscope; 

3.  Take  the  contact’s  BEARING; 

4.  Take  the  contact’s  DISTANCE; 

5.  LOWER  the  periscope; 

6.  Inform  the  contact’s  ANGLE  OE  BOW; 

1 .  CHOOSE  the  contact  to  observe 

Click  on  one  of  these  buttons  to  choose  the  contact  to  observe.  These  three  buttons 
correspond  to  the  three  contacts  outside.  Contact  1  is  the  Erigate  on  your  right  side. 
Contact  2  is  the  Erigate  on  your  left  side,  and  Contact  3  is  the  one  on  your  bow.  In  order 
to  select  the  most  dangerous  contact,  you  will  use  these  3  stopwatches.  As  if  it  happens  to 
the  buttons,  these  three  stopwatches  also  correspond  to  the  contacts  outside.  To  select  the 
correct  contact,  you  will  observe  the  stopwatches  and  click  on  the  button  that  corresponds 
to  the  Stopwatch  with  the  least  remaining  time  (in  this  case,  the  contact  number  3).  The 
remaining  time  in  the  stopwatch  indicates  the  maximum  time  to  observe  the  contact. 

>  ATTENTION:  Always  choose  the  contact  that  has  the  LEAST  remaining 
time.  If  you  choose  the  wrong  contact  to  observe,  you  will  have  to  first 
finish  the  observation  of  the  wrong  contact  and  in  the  next  observation, 
choose  the  correct  contact  to  observe.  If  you  see  a  negative  sign  in  one  of 
the  stopwatches,  it  means  that  you  have  to  choose  this  one  for  the  next 
observation. 
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2.  RAISE  the  periscope 

After  selecting  the  contact,  you  will  have  some  amount  of  time  in  the  correspondent 
stopwatch.  Try  to  RAISE  THE  PERISCOPE  only  when  this  remaining  time  is  close  to 
ZERO.  This  procedure  will  allow  you  to  have  a  better  SCORE  ADJUSTED.  The  SCORE 
ADJUSTED  is  calculated  based  on  the  main  SCORE  and  the  amount  of  exposition  time, 
which  means  the  accumulated  total  time  that  the  periscope  was  up.  The  longer  you  can 
stay  with  the  periscope  down,  the  better  your  ADJUSTED  SCORE  will  be.  You  can 
advance  the  time  by  clicking  the  numbers  from  2  to  9  in  the  main  keypad.  While  holding 
the  key  number  2,  the  time  will  advance  2X  faster.  While  holding  the  key  number  3  the 
time  will  advance  3x  faster,  and  so  on.  When  you  release  the  key,  the  time  will  return  to 
normal  speed.  Use  this  feature  wisely:  it  is  highly  recommended  that  you  use  the  spare 
time  to  study  the  feedback  received  before. 

>  Click  on  this  button  to  RAISE  THE  PERISCOPE.  Eess  than  20  seconds  is 
considered  a  good  timing. 

This  is  the  OUTSIDE  VIEW.  This  is  what  you  see  when  looking  through  the 
periscope.  Here  you  can  read  the  bearing,  the  true  angle  to  where  the  periscope  is 
pointing.  Here  you  can  read  the  distance.  When  the  stadimeter  is  off  the  distance  shown 
is  16000.  When  the  stadimeter  is  on,  the  distance  will  be  displayed  here.  Eater  I  will 
explain  how  to  take  the  contact’s  distance.  Now  let’s  understand  how  to  manipulate  the 
periscope. 

>  To  rotate  the  periscope  to  the  RIGHT  use  the  RIGHT  MOUSE  BUTTON. 

>  To  rotate  it  to  the  EEET,  use  the  EEET  MOUSE  BUTTON. 

>  To  make  the  rotation  EASTER,  press  the  SHIET  button  together  with  the 
mouse. 

>  To  make  the  rotation  SEOWER,  use  the  CONTROE  button. 

>  To  swap  between  high  and  low  ZOOM,  press  the  MIDDEE  MOUSE 
BUTTON. 

3.  Take  the  contact’s  BEARING 

Now  you  are  able  to  take  the  contact’s  BEARING. 

>  Align  the  vertical  line  in  the  middle  of  the  periscope  screen  to  the  center 
of  the  contact. 

>  Click  on  the  BEARING  MARK  button. 

4.  Take  the  contact’s  DISTANCE 

To  take  the  contact’s  distance: 
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>  Rotate  down  the  MOUSE  WHEEL  until  the  horizontal  rear  mast  is 
aligned  to  the  ship’s  WATER  LINE. 

>  Be  sure  that  the  ZOOM  is  applied  (by  clicking  the  MIDLE  MOUSE 
BUTTON). 

>  Click  on  the  DISTANCE  MARK  button. 

5.  LOWER  the  periscope 

Before  LOWERING  the  periscope: 

>  TURN  OEE  the  stadimeter  by  rotating  the  MOUSE  WHEEL  UP; 

>  Return  the  ZOOM  to  LOW  by  clicking  the  MIDLE  MOUSE  BUTTON. 

6.  Inform  the  contact’s  ANGLE  OE  BOW 

The  “Angle  of  Bow”  is  the  angle  from  where  the  contact  can  look  to  your  submarine. 
This  angle  starts  from  his  bow,  to  the  left  or  to  the  right.  So,  to  inform  the  AOB,  you  first 
need  to  inform  the  side  (PORT  or  STARBOARD)  and  then  the  value  of  the  Angle.  In  this 
example,  the  contact  can  see  you  from  his  RIGHT  so  the  side  is  STARBOARD.  This  side 
is  also  the  side  of  the  ship  which  you  can  see.  The  value  of  the  angle  is  around  40°.  So 
the  complete  AOB  is  STBD40.  This  is  the  image  on  the  periscope.  In  this  example,  the 
contact  can  see  you  from  his  LEET  so  the  side  is  PORT.  The  value  of  the  angle  is  around 
70°.  So  the  complete  AOB  is  STBD70.  This  is  how  you  will  see  it  in  the  simulation.  It 
does  not  matter  in  which  side  of  your  bow  the  contact  is.  Pay  attention  only  to  the  side  of 
the  ship  that  you  can  see. 

To  set  the  AoB  value: 

>  Eirst  choose  the  side  (STDB  or  PORT)  and  then,  rotate  the  mouse  wheel 
DOWN  to  increase  or  UP  to  decrease.  Click  in  the  SUBMIT  button  to 
apply  this  value. 

>  Now  you  can  start  a  new  observation  by  choosing  again  the  contact  with 
least  remaining  time  in  the  stopwatch. 

You  will  repeat  this  whole  process  until  one  of  the  contacts  gets  so  close  that  will 
force  you  to  dive,  to  avoid  collision.  This  distance  is  called  GO  DEEP  RANGE.  In  this 
scenario,  it  is  calculated  as  1000  yards.  If  a  contact  is  1200  yards  or  less,  you  are  allowed 
to  dive.  If  it  gets  closer  than  1000  yards,  you  have  to  dive  immediately. 

Congratulations!  You  have  finished  the  tutorial.  Now  you  are  ready  to  the 
experiment.  GOOD  LUCK! 
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VIDEO  TUTORIAL  SCRIPT 
POST-HOC  FEEDBACK  GROUP 


Welcome  to  the  SUBMARINE  TACTICAL  TRAINING  SYSTEM,  and  thank  you 
for  applying  to  this  experiment.  This  is  the  MAIN  SCREEN.  In  the  LEFT  PANEL  you 
will  see  INTERIOR  OF  THE  SUBMARINE,  if  the  periscope  is  DOWN  or  the 
OCEANIC  ENVIRONMENT  outside  the  submarine,  if  the  periscope  is  UP.  The 
CENTER  PANEL  is  called  INPUT  PANEL  and  it  will  be  used  to  control  the  periscope, 
give  orders  and  provide  information  about  the  observations.  The  RIGHT  PANEL  is 
called  the  FEEDBACK  PANEL  and  it  will  be  used  to  show  the  scores. 

One  observation  consists  of  this  sequence: 

1.  CHOOSE  the  contact  to  observe; 

2.  RAISE  the  periscope; 

3.  Take  the  contact’s  BEARING; 

4.  Take  the  contact’s  DISTANCE; 

5.  LOWER  the  periscope; 

6.  Inform  the  contact’s  ANGLE  OF  BOW; 

1 .  CHOOSE  the  contact  to  observe 

Click  on  one  of  these  buttons  to  choose  the  contact  to  observe.  These  three  buttons 
correspond  to  the  three  contacts  outside.  Contact  1  is  the  Frigate  on  your  right  side. 
Contact  2  is  the  Frigate  on  your  left  side,  and  Contact  3  is  the  one  on  your  bow.  In  order 
to  select  the  most  dangerous  contact,  you  will  use  these  3  stopwatches.  As  if  it  happens  to 
the  buttons,  these  three  stopwatches  also  correspond  to  the  contacts  outside.  To  select  the 
correct  contact,  you  will  observe  the  stopwatches  and  click  on  the  button  that  corresponds 
to  the  Stopwatch  with  the  least  remaining  time  (in  this  case,  the  contact  number  3).  The 
remaining  time  in  the  stopwatch  indicates  the  maximum  time  to  observe  the  contact. 

>  ATTENTION:  Always  choose  the  contact  that  has  the  LEAST  remaining 
time.  If  you  choose  the  wrong  contact  to  observe,  you  will  have  to  first 
finish  the  observation  of  the  wrong  contact  and  in  the  next  observation, 
choose  the  correct  contact  to  observe.  If  you  see  a  negative  sign  in  one  of 
the  stopwatches,  it  means  that  you  have  to  choose  this  one  for  the  next 
observation. 

2.  RAISE  the  periscope 

After  selecting  the  contact,  you  will  have  some  amount  of  time  in  the  correspondent 
stopwatch.  Try  to  RAISE  THE  PERISCOPE  only  when  this  remaining  time  is  close  to 
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ZERO.  This  procedure  will  allow  you  to  have  a  better  SCORE  ADJUSTED.  The  SCORE 
ADJUSTED  is  calculated  based  on  the  main  SCORE  and  the  amount  of  exposition  time, 
which  means  the  accumulated  total  time  that  the  periscope  was  up.  The  longer  you  can 
stay  with  the  periscope  down,  the  better  your  ADJUSTED  SCORE  will  be.  You  can 
advance  the  time  by  clicking  the  numbers  from  2  to  9  in  the  main  keypad.  While  holding 
the  key  number  2,  the  time  will  advance  2X  faster.  While  holding  the  key  number  3  the 
time  will  advance  3x  faster,  and  so  on.  When  you  release  the  key,  the  time  will  return  to 
normal  speed. 

>  Click  on  this  button  to  RAISE  THE  PERISCOPE.  Eess  than  20  seconds  is 
considered  a  good  timing. 

This  is  the  OUTSIDE  VIEW.  This  is  what  you  see  when  looking  through  the 
periscope.  Here  you  can  read  the  bearing,  the  true  angle  to  where  the  periscope  is 
pointing.  Here  you  can  read  the  distance.  When  the  stadimeter  is  off  the  distance  shown 
is  16000.  When  the  stadimeter  is  on,  the  distance  will  be  displayed  here.  Eater  I  will 
explain  how  to  take  the  contact’s  distance.  Now  let’s  understand  how  to  manipulate  the 
periscope. 

>  To  rotate  the  periscope  to  the  RIGHT  use  the  RIGHT  MOUSE  BUTTON. 

>  To  rotate  it  to  the  EEET,  use  the  EEET  MOUSE  BUTTON. 

>  To  make  the  rotation  EASTER,  press  the  SHIET  button  together  with  the 
mouse. 

>  To  make  the  rotation  SEOWER,  use  the  CONTROE  button. 

>  To  swap  between  high  and  low  ZOOM,  press  the  MIDDEE  MOUSE 
BUTTON. 

3.  Take  the  contact’s  BEARING 

Now  you  are  able  to  take  the  contact’s  BEARING. 

>  Align  the  vertical  line  in  the  middle  of  the  periscope  screen  to  the  center 
of  the  contact. 

>  Click  on  the  BEARING  MARK  button. 

4.  Take  the  contact’s  DISTANCE 

To  take  the  contact’s  distance: 

>  Rotate  down  the  MOUSE  WHEEE  until  the  horizontal  rear  mast  is 
aligned  to  the  ship’s  WATER  EINE. 

>  Be  sure  that  the  ZOOM  is  applied  (by  clicking  the  MIDEE  MOUSE 
BUTTON). 


81 


>  Click  on  the  DISTANCE  MARK  button. 


5.  LOWER  the  periscope 

Before  LOWERING  the  periscope: 

>  TURN  OLE  the  stadimeter  by  rotating  the  MOUSE  WHEEL  UP; 

>  Return  the  ZOOM  to  LOW  by  clicking  the  MIDLE  MOUSE  BUTTON. 

6.  Inform  the  contact’s  ANGLE  OE  BOW 

The  “Angle  of  Bow”  is  the  angle  from  where  the  contact  can  look  to  your  submarine. 
This  angle  starts  from  his  bow,  to  the  left  or  to  the  right.  So,  to  inform  the  AOB,  you  first 
need  to  inform  the  side  (PORT  or  STARBOARD)  and  then  the  value  of  the  Angle.  In  this 
example,  the  contact  can  see  you  from  his  RIGHT  so  the  side  is  STARBOARD.  This  side 
is  also  the  side  of  the  ship  which  you  can  see.  The  value  of  the  angle  is  around  40°.  So 
the  complete  AOB  is  STBD40.  This  is  the  image  on  the  periscope.  In  this  example,  the 
contact  can  see  you  from  his  LEET  so  the  side  is  PORT.  The  value  of  the  angle  is  around 
70°.  So  the  complete  AOB  is  STBD70.  This  is  how  you  will  see  it  in  the  simulation.  It 
does  not  matter  in  which  side  of  your  bow  the  contact  is.  Pay  attention  only  to  the  side  of 
the  ship  that  you  can  see. 

To  set  the  AoB  value: 

>  Eirst  choose  the  side  (STDB  or  PORT)  and  then,  rotate  the  mouse  wheel 
DOWN  to  increase  or  UP  to  decrease.  Click  in  the  SUBMIT  button  to 
apply  this  value. 

>  Now  you  can  start  a  new  observation  by  choosing  again  the  contact  with 
least  remaining  time  in  the  stopwatch. 

You  will  repeat  this  whole  process  until  one  of  the  contacts  gets  so  close  that  will 
force  you  to  dive,  to  avoid  collision.  This  distance  is  called  GO  DEEP  RANGE.  In  this 
scenario,  it  is  calculated  as  1000  yards.  If  a  contact  is  1200  yards  or  less,  you  are  allowed 
to  dive.  If  it  gets  closer  than  1000  yards,  you  have  to  dive  immediately. 

Congratulations!  You  have  finished  the  tutorial.  Now  you  are  ready  to  the 
experiment.  GOOD  LUCK! 
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APPENDIX  G.  POST-TASK  SURVEY 


GODEX  Task 
Post  Task  Survey 

Subject  number:  Date: 

1.  During  the  task,  what  was  your  strategy  to  avoid  "losing  the  stopwatch"  of  a  contact? 

a.  I  had  no  specific  strategy. 

b.  I  waited  until  the  last  moment  to  observe  a  contact  to  avoid  "Exposition 
Penalty". 

c.  I  waited  until  the  last  moment,  taking  into  consideration  the  stopwatch  of  the 
next  contact  to  observe. 

d.  I  observed  the  contacts  as  soon  as  I  finished  the  previous  one. 

e.  Other 


2.  Do  you  think  that  your  performance  improved  during  the  course  of  the  experiment? 
Please  comment. 


3.  If  you  are  (or  was)  a  submariner,  how  do  you  compare  this  simulation  to  other  that 
you  have  saw  during  your  career.  Please  comment. 

a.  Significantly  Worse 

b.  Worse 

c.  Similar  - 

d.  Better  - 

e.  Significantly  Better  - 

f.  - 

4.  If  you  received  feedback  during  the  runs,  please  rank  the  type  of  feedback  from  least 
useful(l)  to  most  useful(5): 

a.  Auditory:  1  2  3  4  5 

b.  Text  (Pop-up  Messages):  1  2  3  4  5 

c.  Text  (Message  Log):  1  2  3  4  5 

d.  Color  (Pop-up  Messages):  1  2  3  4  5 

e.  Scores:  1  2  3  4  5 
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APPENDIX  H.  DETAILED  STATISTICAL  RESULTS 


IMatched  Pairs  GROUP=RTF 


Difference:  ADJ  SCORE  RUN2-ADJ  SCORE  RUN1  Difference:  ADJ  SCORE  RUN3-ADJ  SCORE  RUN1 


o 

U 


a 

< 


z 

u 


15000- 


10000- 


Q  5 


o 

< 


5 


-10000- 


-15000- 


ADJ  SCORE  RUN2 

*•  •  • 

15000-1 

LU 

QC 

O 

y  10000- 

O 

< 

2  5000- 

ADJ  SCORE  RUNS 

• 

• 

. . . . . 

. r . . 

. ^ . * . 

. ^ • . 

ADJ  SCORE  RUN1 

- 1 - 1 - 1—1 - 1 - 1 - 1 - 1 - 1 - 1 - 1 - 1 - 1 

si 

u  “ 

!✓» 

1  -5000- 

4> 

%  -10000- 
It 

5 

-15000- 

% 

• 

ADJ  SCORE  RUN1 

— 1—1 — 1 — 1 — 1 — 1 — 1 — 1 — 1 — 1 — 1 — 1 — 1 

2000  4000  6000  8000  1000012000 
Mean;  (ADJ  SCORE  RUN2*ADJ  SCORE 
RUN1)/2 


2000  4000  6000  8000  1000012000 
Mean;  (ADJ  SCORE  RUN3*ADJ  SCORE 
RUN1)/2 


ADJ  SCORE  RUN2 

47532  t-Ratio 

3.587511 

ADJ  SCORE  RUN1 

3494.11  DF 

16 

Mean  Difference 

1259.09  Prob  >  |t| 

0.0025* 

Std  Error 

350.965  Prob  >  t 

0.0012* 

Upper  95% 

2003.1  Prob  <  t 

0.9988 

Lower  95% 

515.078 

N 

17 

Correlation 

0.81413 

ADJ  SCORE  RUN3 

586929  t-Ratio 

3.982161 

ADJ  SCORE  RUN1 

3494.11  DF 

16 

Mean  Difference 

2375.18  Prob  >  |t| 

0.0011* 

Std  Error 

596.455  Prob  >  t 

0.0005* 

Upper  95% 

3639.61  Prob<t 

0.9995 

Lower  95% 

1110.75 

N 

17 

Correlation 

0.73933 

Difference:  ADJ  SCORE  RUN3-ADJ  SCORE  RUN2 


o 

u 


o 

< 


z 

u 


Q  5 


o 

< 


5 


15000- 

10000- 

5000- 

0- 

-5000-1 

-10000- 


ADJ  SCORE  RUN3 


-15000 


ADJ  SCORE  RUN2 

“T 


1 — ' — I — ' — I — ' — I — ' — r 
2000  4000  6000  8000  1000012000 
Mean;  (ADJ  SCORE  RUN3*ADJ  SCORE 
RUN2)/2 


ADJ  SCORE  RUN3 

586929  t-Ratio 

2.312486 

ADJ  SCORE  RUN2 

47532  DF 

16 

Mean  Difference 

1116.09  Prob  >  |t| 

0.0344* 

Std  Error 

482.636  Prob  >  t 

0.0172* 

Upper  95% 

213923  Prob  <t 

0.9828 

Lower  95% 

92.9465 

N 

17 

Correlation 

0.81839 

85 


[Matched  Pairs  GROUP^Pt^ 


0  2000  4000  6000  800010000  14000 

Mean:  (ADJ  SCORE  RUN2*ADJ  SCORE 
RUN1)/2 


2000  4000  6000  800010000  14000 

Mean:  (ADJ  SCORE  RUN3*ADJ  SCORE 
RUN1)/2 


ADJ  SCORE  RUN2 

4899.35  t-Ratio 

6.885527 

ADJ  SCORE  RUN3 

6178.79  t-Ratio 

7.143197 

ADJ  SCORE  RUN1 

2798.6  DF 

16 

ADJ  SCORE  RUN1 

2798.6  DF 

16 

Mean  Difference 

2100.75  Prob  >  |t| 

<.0001* 

Mean  Difference 

3380.18  Prob  >  |t| 

<■0001* 

Std  Error 

305.097  Prob  >  t 

<.0001* 

Std  Error 

473203  Prob  >  t 

<.0001* 

Upper  95% 

2747.53  Prob  <  t 

1.0000 

Upper  95% 

438333  Prob  <  t 

1.0000 

Lower  95% 

1453.97 

Lower  95% 

2377.04 

N 

17 

N 

17 

Correlation 

0.79858 

Correlation 

0.6259 

Difference:  ADJ  SCORE  RUN3-ADJ  SCORE  RUN2 


10000 


o 

u 


O 

< 


Z 

3 


< 


5 


5000- 


0- 


-5000- 


-10000 


— I - 1 - 1 - 1 - 1 - 1 — 

2000  4000  6000  8000  1  0000  1  4000 

Mean:  (ADJ  SCORE  RUN3+ADJ  SCORE 
RUN2)/2 


ADJ  SCORE  RUN3 

6178.79  t-Ratio 

3.479882 

ADJ  SCORE  RUN2 

489935  DF 

16 

Mean  Difference 

127943  Prob  >  |t| 

0.0031* 

Std  Error 

367.666  Prob  >  t 

0.0015* 

Upper  95% 

205885  Prob  <  t 

0.9985 

Lower  95% 

500.017 

N 

17 

Correlation 

0.79519 

86 


LIST  OF  REFERENCES 


About.  (2016).  Retrieved  from  http://www.blender.org/about 

Anderson,  J.  R.  (1987).  Skill  acquisition:  Compilation  of  weak-method  problem 
solutions.  Psychological  Review,  94,  129-210. 

Area  Militar.  (2011).  Fragata  Classe  Greenhalgh  (tipo  Type  22)  [Website].  Retrieved 
from  http://www. areamilitar.net/directorio/NAV. aspx?NN=26 

Astwood,  R.,  Van  Buskirk,  W.,  Cornejo,  J.  &  Dalton,  J.  (2008).  The  impact  of  different 
feedback  types  on  decision-making  in  simulation  based  training  environments. 
Proceedings  of  the  52nd  Annual  Meeting  of  the  Human  Factors  and  Ergonomics 
Society,  New  York,  NY. 

Bastos,  R.  S.  (2014).  Development  of  a  web-based  periscope  simulator  for  submarine 
officer  training  (Master’s  thesis).  Retrieved  from  Calhoun: 
http://calhoun.nps  .edu/handle/ 1 0945/43873 

Branson,  R.K.,  Rayner,  G.T.,  Cox,  J.L.,  Furman,  J.P.,  King,  F.J.,  Hannum,  W.H. 

(1975).  Interservice  procedures  for  instructional  systems  development.  (Vols.  1-5) 
TRADOC  Pam  350-30,  NAVEDTRA  106A.  Et.  Monroe,  VA:  U.S.  Army 
Training  and  Doctrine  Command. 

Buff,  W.  E.,  &  Campbell,  G.  E.  (2002).  What  to  do  or  what  not  to  do?:  Identifying  the 
content  of  effective  feedback.  Proceedings  of  the  Human  Factors  and 
Ergonomics  Society  Annual  Meeting,  46,  2074-2078. 
http://doi.org/10.1177/154193120204602517 

Cohen,  V.  B.  (1985).  A  reexamination  of  feedback  in  computer-based  instruction: 

Implications  for  instructional  design.  Educational  Technology,  25(1),  33-31. 

Deane,  D.  H.,  Hammond,  K.  R.,  &  Summers,  D.  A.  (1972).  Acquisition  and  application 
of  knowledge  in  complex  inference  tasks.  Journal  of  Experimental  Psychology, 
92(1),  20-26.  Retrieved  from  http://dx.doi.org/10.1037/h0032162 

Eerguson,  J.  H.  (2014).  Canadian  submariners  in  Europe,  1915-1917.  In  Through  a 

Canadian  periscope:  The  story  of  the  Canadian  submarine  service  (2nd  Edition), 
pp.  96-96.  Toronto,  Canada:  Dundum  Group. 

Gist,  M.  E.,  &  Mitchell,  T.  R.  (1992).  Self-Efficacy:  A  theoretical  analysis  of  its 

determinants  and  malleability.  Academy  of  Management  Review ,  17,  183-211. 

Hammond,  K.  R.  (1971).  Computer  Graphics  as  an  Aid  to  Eearning.  Science,  772(3986), 
903-908.  American  Association  for  the  Advancement  of  Science.  Retrieved  from 
http  ://science .  sciencemag.org/content/ 172/3986/903 

87 


Hawkins,  J.  (2013,  February  24).  Ocean  surface  effects  in  Unity  [Blog  post].  Perth,  WA: 
ScrawBlog.  Retrieved  from  https://scrawkblog.eom/2013/02/24/ocean-renderer- 
for-unity 

Ilgen,  D.R.,  Fisher,  C.D.,  &  Taylor,  M.T.  (1979).  Consequences  of  individual  feedback 
on  behavior  in  organizations.  Journal  of  Applied  Psychology,  64,  349-371. 

Kluger,  A.N.  &  DeNisi,  A.  (1996).  The  effects  of  feedback  interventions  on  performance: 
A  historical  review,  a  meta-analysis,  and  a  preliminary  feedback  intervention 
theory.  Psychological  Bulletin,  119,  254-284. 

Kozlowski,  S.W.J.,  Toney,  R.J.,  Mullins,  M.E.,  Weissbein,  D.A.,  Brown,  K.G.  &  Bell, 
B.S.  (2001).  Developing  adaptability:  A  theory  for  the  design  of  integrated- 
embedded  training  systems.  In  E.  Salas  (Ed.),  Advances  in  human  performance 
and  cognitive  engineering  research  (pp.  59-123).  Amsterdam:  JAEElsevier 
Science. 

Eandsberg,  C.  R.,  Van  Buskirk,  W.  E.,  &  Astwood,  R.  S.  (2010).  Does  feedback  type 
matter?  Investigating  the  effectiveness  of  feedback  content  on  performance 
outcomes.  Proceedings  of  the  Human  Factors  and  Ergonomics  Society  Annual 
Meeting,  54(21),  2304-2308.  Retrieved  from 
http://pro.sagepub.com/content/54/27/2304 

Mack,  S.  (2003).  Perisher:  Submarine  command  training  in  the  Royal  Navy.  Undersea 
Warfare,  Spring.  Retrieved  from  http://www.public.navy.mil/subfor/ 
underseawarfaremagazine/Issues/Archives/issue_18/perisher.htm 

Michael,  D.  R.,  &  Chen,  S.  E.  (2005).  Serious  games:  Games  that  educate,  train,  and 
inform.  Independence,  KY:  Cengage  Eeaming  PTR.. 

NATO.  (2002).  Multinational  submarine  and  antisubmarine  exercise  manual  (MXP-ID, 
Vol.  1).  Washington,  DC:  Author. 

Perisher:  The  British  Submarine  Commanders  Qualifying  Course.  (2007,  October  27). 
Retrieved  from  http://forums .ubi.com/showthread.php/650328-PERISHER-The- 
British-Submarine-Commanders-Qualifying-Course-Eorums 

Perlin,  K.  (2002,  July).  Improving  noise.  In  ACM  Transactions  on  Graphics  (TOG), 
27(3),  681-682. 

Sadler,  D.  R.  (1989).  Eormative  assessment  and  the  design  of  instructional  systems. 
Instructional  Science,  18(2),  119-144.  Retrieved  from 
http :  //link,  springer  .com/10. 1007/BP00117714 

SAS  Institute.  (2015),  JMP  (Version  12.0)  [Computer  software],  Cary,  NC.  Retrieved 
from  http://www.jmp.com/en_us/offers/free-trial.html 


88 


Smither,  J.W.,  Wohlers,  A.J.,  &  London,  M.  (1995).  A  field  study  of  reactions  to 
normative  versus  individualized  upward  feedback.  Group  Organization 
Management,  20,  61-89. 

Summers,  D.  A.,  Taliaferro,  J.  D.,  &  Fletcher,  D.  J.  (1970).  Judgment  policy  and 

interpersonal  learning.  Behavioral  Science,  15(6),  514-521.  Retrieved  from 
http://doi.wiley.eom/10.1002/bs.3830150605 

Todd,  F.  J.,  &  Hammond,  K.  R.  (1965).  Differential  feedback  in  two  multiple-cue 

probability  learning  tasks.  Behavioral  Science,  10(A),  429-435.  Retrieved  from 
http  ://doi .  wiley .  com/ 10. 1002/bs. 38301 00406 

Unity,  (n.d.).  Retrieved  June  16,  2016,  from  http://unity3d.com/unity 

White,  D.  S.  H.  (1993,  October).  The  submarine  command  course.  The  Naval  Review, 

81(A),  362-365.  Retrieved  from  http://www.naval-review.com/issues/1990s/1993- 
4.pdf 


89 


THIS  PAGE  INTENTIONALLY  LEET  BLANK 


90 


INITIAL  DISTRIBUTION  LIST 


1 .  Defense  Technical  Information  Center 
Ft.  Belvoir,  Virginia 

2.  Dudley  Knox  Library 
Naval  Postgraduate  School 
Monterey,  California 


91 


